In flight, no real math is used; instead, a bunch of thumb-rules is used. In in-flight related questions, you will not be tested for math but for these simplified rules. You'll be taught the rules at some point in your training.
There is indeed some math in questions regarding pre-flight preparation, but it is mostly limited to addition/subtraction, and using a calculator is allowed.
Conversion of time and fuel is best helped with a
Flight Computer. It is allowed on the test; in fact it is a MUST USE thing as it calculates
quicker than an electronic calculator.
Regarding the memory for numbers, the longest ATC clearance you're legally required to accept is 3 elements (e.g., altitude + speed + heading instruction) - I don't believe anyone would have problem remembering three numbers.