Stick with the flying, but ditch the instructor!
I took up flying to get over a fear of flying caused by a number of helicopter incidents in the military which kept me from flying in anything for 25 years. To say I was a nervous student would be an extreme understatement! When my first instructor went off to fly big-uns, a week after I had bent an aircraft in a landing accident on my third solo, I was given another instructor, who although a really nice guy and excellent pilot, was very young and wasn't getting anywhere with neurosis-ridden me.
Luckily, I was learning with a very professional outfit who realised this and substituted him for the assistant CFI and another, older AFI, who double-teamed me. I'd fly with one to our main training field and back with the other, and all of a sudden things clicked. Instead of dreading flying, I began to enjoy it. One instructor hammered home the technique in a persistant, but patient way, the other gave me confidence and made me realise I could do it. Had it not been for these two I would never have got my licence.
Moral of the story- if you aren't 100% happy with your instructor you won't progress.