I don't remember there being much physics (except of the most elementary kind) required for the PPL. There doesn't seem to be much either in the JAR CPL exams which is a pity because I'm a physicist and I'm studying for them at present.
There is some simple maths required (cross-multiplication, and similar, plus some simple trig) but nothing as difficult as calculus. At least I've not come across any yet.
Most University Entrance qualifications (in the UK at least) say things like "is normally required" "should have" etc. If you talk to them your will probably find there is a way forward. If they are short of candidates they might suggest a suitable bridging course to bring you up to speed on whatever they feel is lacking.
Good Luck.