You need to be able to grasp the concept, rather than worrying about the calculation methods. It's not really mathematics beyond mid-high school level. You have to think in terms of miles, feet, and degrees of the compass or angle. Speed and time calculations, etc.
Being able to understand how you get from Here to There without too much fancy equipment is important. In the same way that making change in the store shouldn't require a calculator, as it seems to so often.
And even if you eventually end up in a big jet where someone else works stuff out for you, you have to learn the concepts at the very beginning as you earn your Private. It's not so tough, and you should be just fine.