I did Physics at university and realised within the first week that the maths was starting from a position which was way above my level at the time. (I did not attend the best of secondary schools). I considered my options of changing courses, quitting etc., but eventually decided to stick it out and just work hard to catch up. I can honestly say I left 90% of all lectures during that degree, not really having a scobbies what had happened. Yet I graduated with a first class honours degree from the best place in the country to study Physics at the time. I guess I saw it as a challenge to overcome. Good luck.