i_m_p,
I was military and now GA, so have never experienced the airline job, but followed my dream as a young fella and would never change it.
Your situation sounds like one of those where if you don't give it a go, you will always be wondering 'what if?'
You're young enough that if it doesn't work out, you have many options anyway, and finishing the high school maths and physics will no doubt be beneficial in any case.
You don't have to start tomorrow, but how about early next year - there's a time frame, you've had a bit of a break after the uni, and it's a step on the way.
I've also read on various threads people talking about shorter aviation-connected maths and physics courses that may allow you to knock it over quicker than you think.
Good luck.