Just a thought, but have you figured out which aspect of the Nav exercises are causing your anxiety? You mention having encountered this before during pre-solo training, but you 'stuck with it'. Did the anxiety lessen as you passed the first solo hurdle? A bit of anxiety may be 'normal' when training for something like a PPL, but if its the flying itself that's causing it, there may be something else behind it.
I can understand you wanting to avoid another stressful occupation next to your job, but even though it does cause some sort of stress, being completely away from the job and focusing on something entirely different may be good for you in the long run. Something else you could try is finding someone with a PPL and going along for an expensive lunch at some other airfield. It might give you another view of what's possible as in contrast to the navigation training, bimbling along on a nice day, visiting another airfield and enjoying a decent lunch may provide a different perspective on the issue as its a completely different experience than sweating over courses and timings with an instructor breating down your neck.
I hope you manage to make a decision about this that suits you.