Do it.
If you can face the next 20+ years of your working life wondering if you should have done it or not, then maybe it's not worth the risk.
Wouldn't it be better to think, when picking up the pension "I tried it, gave it my best shot, and it didn't work out for me" than "I really wish I'd......"