Thanks folks, that pretty much goes with what I was thinking.
But what of actually flying under IFR, rather than 'stealth mode' VFR in IMC? The fundamental question is can you actually get away with flying IFR, in controlled airspace to or from a 'proper' airport, in a Permit aircraft?