Right, here's my personal experience.
I left the RAN (non-aviation role) in 2005, aged 30, worked for Surveillance Australia doing Coastwatch ops in the back of BN2B Islanders and Dash 8's. Moved on to AeroRescue after that and the guys convinced me to do my RAAus cert. Got bored of having to use runways so did a trial flight in a heli and from that commenced my CPL(H) aged 34. Finished that on the 18th July 2011. At 36 I was on my own in the big wide world of rotary aviation.
I turned 39 in March this year and celebrated the first 6 months of running my own tourism helicopter business.
Mate, you're a diver. You wear a big watch and you're a stubborn bastard.
Don't let some armchair pilot tell you that you can't do it because if I can, you sure as hell can!! It'll be hard but you've dealt with worse and there's going to be some serious challenges but you're already used to that as well.
There's a lot of disgruntled guys here because they started young and they've done their time, this will be new to you and you'll love it for years to come.
Regret. Don't let that get to you. Be one of those guys who, at the end of the line, say's "hell yeah I gave it a go" instead of one of those blokes who sits there alone at the bar going "I wonder what would've happened if I did that"
I'm broke, don't sleep well most nights because I'm freaked out about making budget at the end of each month but I love flying and I love what I do and I'm bloody amazed at what I've achieved in 5 years.
It's not that it can't be done, it's how you will make it happen!