That rather depends on what degree. If by "degree level maths" you mean you're a maths grad then you can stop worrying - pilots need arithmetic and some trig, but that's about it. If you mean "I have maths from an engineering/science degree" then the same applies. But if you mean "I had good enough maths GCSEs to get me onto a Media Studies, fine art or sociology degree" then have a look at a decent book like the ones recommended in this thread and form your own opinion!
PDR