Hi Dan,
I have been a commercial pilot for 18 years now, but I remember my PPL days.
You definitely need one instructor whom you know well, not pot-luck on the day. You need a sympathetic teacher and to get used to each other and build a relationship. Speak with the flying club boss. You are a customer and you are entitled to decide who you pay your money to. Just tell them you are underconfident and need a really good, patient, instructor.
Nav work can be intimidating - especially if your instructor is passively or actively aggressive about mental maths. Some can be like this, either deliberately or accidentally and destroy your confidence. I can work out things like the formula for the area of a hexagon in my head from first principles, but if an authority figure is barking mental maths questions at me I might not be able to add two and two together !
Master the CRP-5. I found the visual way of working out the drift etc to be very useful. Also use the Trevor Thom books.
Fly over areas you know well if possible. Spotting ground features and towns etc and knowing where you are is much easier this way. Having said that, I was on an early NAVEX once and getting slightly flustered trying to find Kidlington aerodrome, (Oxford). I thought I should be there but couldn’t see it. My instructor casually asked if I could see an aerodrome anywhere. It was right in front of me, under my nose !!. My navigation had been spot-on, but there was so much going on I was swamped. On my qualifying cross country I confidentally told ATC I was crossing the M4, only to call back 5 mins later to report I was crossing it now - I had been mistaken earlier!
I would try to complete your PPL if you can find a sympathetic instructor - you will have wasted a lot of money if you give up now.
If navigation is really not for you, how about glider flying? Obviously you still have to navigate to an extent, and avoid certain airspace, but you are much freer to simply enjoy flying for flyings sake. Much cheaper too.
Good luck !