you must (IFR) or can (VFR) set 1013 as your reference pressure setting
I would not consider setting 1013 optional, even if VFR. ATC will issue clearances and such, and will expect your altitude reports to be a FL instead of an altitude, once you are above the TA. Plus there's the quandrantical rule (UK) or semicircular rule (elsewhere) that requires you to fly VFR at specific FLs above 3000' AGL. If you fly a FL according to those rules, but have QNH still set on the altimeter you might be in for some serious surprises.
Not to say that I adjust my altimeter every time I go through the layer though. When I'm doing aerobatics training I typically request a block between "3000 feet and flight level 55" (because of the airspace above and below my usual training area). I leave the altimeter on QNH, and read the FLs off the transponder. It's a bit awkward to adjust the altimeter twice during a reverse half cuban...