No problems at all from the grammatical point of view. "Yesterday" is not the time of the habitual /permanent event but of the one that stopped it. As always, however, it depends what meaning you want to express. "used to" expresses a habitual fact about the past which is no longer true. If you stopped smoking two years ago, then you can certainly feel fairly sure that the habitual action is no longer true - the previous and current situations are different -
I used to smoke but I quit last year, and I feel much better for it- I certainly wouldn't want to start again.
If on the other hand it only happened recently, you can't necessarily "guarantee" that the habitual/permanent event really has finished. If you only quit smoking yesterday you can't be sure that you'll have the will-power to continue. So to say
I used to smoke but I quit yesterday sounds a bit odd - or at least over-optimistic
But as always, it's a matter of context and speaker intention - not of grammar. The example
There used to be a building there but they knocked it down yesterday sounds much less odd, because, clearly, there's no chance of the original building suddenly reappearing. The previous situation has changed irrevocably.