Choppertop... much the same situation. Have an existing (diverse) business and put my PPL(H) VAT through on the grounds that eventually it will progress to another source of income.
Received much the same advice as yourself from my accountant.
Got a notice informing me of a routine VAT inspection; at which point I expected to be told I would have to repay it all.
The chap accepted it at face value, although said they would come back out in 3 years to see how I am getting on. Apparently they accept that element of the VAT returns on an "intending trader" basis.
By all accounts though, not all inspectors are as accommodating!