What you probably mean is "you don't have to fly at a rounded FL according to the quadrantal/semi-circular rule" if VFR.
No I mean (in the UK) you can set the local altimeter setting and fly around at an altitude. For example if I am VFR from X to Y I can call up ATC and request something and report my altitude as Altitude 7000 on QNH 1002 for example. There is no requirement to set 1013 above the TA for VFR flight.
However it is wise to. I heard some chap on the radio on his way to the Channel Islands the other day at 4000' QNH. The controller advised him that he may infringe Q41 which has a base of FL35 to which he replied that on that altimeter setting he should be below FL35...."shouldn't he?". Then the controller responded that she'd get back to him after she had calculated it. Would have been a lot easier for him to just set 1013 on the altimeter and remain below FL35.
Of course around the LTMA the airspace is delimited by altitude (London QNH), elsewhere it might be flight level. It does get horribly complicated especially when you get say an airway which is defined as "
FL65 (Minimum altitude 5500')".