As other have said, get a new instructor. Its not cheap to learn to fly and it should be fun, or as fun as it can be when you're being taught something as complex as this.
As for in-flight maths, you should be being taught to handle the aeroplane before being expected to do calculations such as fuel required to reach a diversion.
I got to my test in minimum time and I'm sure my time with a flight simulator was key to this. I didn't use the simulator to learn to handle the plane (they're not much use for this), I used it to expand my capacity and learn my checks and practice my radio calls. The idea was to simulate the mental loading of flying the plane and keeping the needles pointing in the right direction (alt,ASI,compass) whilst practicing making the right call at the right time, or doing the mental calculations for a diversion. This practice meant that in the real thing, my brain was having to adjust to doing a familiar procedure in a noisy box, rather than translating something that I had only done in the classroom.
Now I'm a seasoned skygod
I have pretty awful mental arithmetic but I've got very good at guestimating my way around the sky. I only fly VFR and always plan any cross-countries to leave me with and hour's fuel at my destination, but beyond that I'm very lazy. This doesn't mean I don't know where I am, but it does mean that when London Information asks me for an ETA to the FIR Boundary when I'm somewhere near Beachy Head, my estimates on the fly are very inaccurate. The potential for big cockups is when flying over water and when crossing the Channel I'm very anal about getting all the maths right before taking off.