Don't give up - go somewhere else , you are learning to fly not do maths .
Yes there is some maths element in getting your PPL but a bit of dead reckoning , simple map reading , visual location awareness and you'll crack it.
I don't remember getting much to do with maths in the air with my instructors , may be a few headings changes , but if I wasn't happy with anything or the way it was going I made it clear.
If you walk away from the school it can look bad on them , so they are expected to get the tuition across , if you don't like the attitude of the instructor just take your business somewhere else.
I tried another flying school when trying to decide on best flying school for my training , took an instant dislike to the instructor on the first lesson , bad attitude , anyway they didn't get anymore money or business off me.
Very happy with my final choice of school and the instructors were brilliant.