Do it man;
I fly for a living and still fly in my spare time, I also sail. You will always find it easier to get time to fly rather than to sail.
You will be amazed when you hear your colleagues being incredibly patient and be able to take the p*** unmercifully when you get back to work.
Don't think I could ever have parted from a Pilatus B4 though.