Pat,
Obviously, as others have stated, such situations are hard to judge objectively in a forum like this with one side of the story heard.
But nevertheless: I don't see how it is some instructor's call to judge whether you're learning too slow and should just leave it - and I find it unprofessional of an instructor to imply this.
That's entirely your call and you need to be happy with your progress or even with flying in general! It's not so much about getting there (to the license) as fast as possible. As long as you're enjoying flying (with an instructor to your right or not), everything is fine!
I believe some of the best pilots out there have taken their time to learn their craft.
And I'd suggest to change the instructor, too. You're obviously not quite on the same page.