In the UK where IFR and VFR merge into some strange condition when in Class G, the rule you have to watch is the 500' rule, so assuming over the sea, then 500' would be a safe minimum. You could in theory go lower, so long as you don't end up within 500' of anyone.....Even this may not be applicable if you're landing, I can't remember.