While it's true that Physics plays a larger part in the studies (it is relevant for at least two subjects), you by no means need to know everything, and your school will typically provide the basic information you need. Oxford do CDs for the relevant basics anyway. The maths? A little bit of algebra, be able to manipulate formulae. You don't need calculus, except to understand the basics behind many processes (e.g. Inertial Nav), plane geometry, SIN, COS, TAN, etc.
I wouldn't be putting obstacles in the way. Just do it - you may surprise yourself. We have students of 53 passing their exams.