Another way to think about this, is to think that learning to fly is what you are doing. I mean don't worry too much about your progress, and to just get on and do what you can afford/have the time for. Eventually you will qualify and that will enable you move on to another stage of learning about flying (and learning about yourself).
So don't fret about it. Talk to your current instructor (that's a very good idea) and decide what you will emotionally feel comfortable with. If moving feels good, then do that. If staying put feels good, then do that. Only you can know.
You are buying a leisure activity. Treat it the same way you would any other leisure activity. Relax.
(Obviously I mean all this in the context of someone who actually is learning to fly for fun. Not a career.)