If it's something that you want to do, truly from your heart then do it. You only have one life, what's the point in living it with regrets. At least you have a fantastic career to fall back on. I work with 2 pilots, one is a Doctor and one a Dentist. They still work outside of flying. Go for it.