Sam, good point. But this could be mitigated by having unambiguous VFR reporting points, easy to distinguish in the field, on the map and in the GPS. Road, railway, river crossings, edges of towns, that sort of thing.
I used to think that would be a problem. Now I find most IFR "intersections" can actually be located visually with good accuracy. Occasionally it might be a little extra challenge. The type of waypoints we usually prefer for VFR navigation can usually be found close to the IFR waypoint as well, to aid you in locating the IFR waypoint. So by now I almost always use IFR waypoints for VFR navigation; it makes both programming the GPS, filing the ATS flightplan and communicating with ATC so much easier!
Better still, most VFR pilots think IFR waypoints can't be located so they don't use them, meaning less crowded skies for me...
I tried programming a full VFR flight with half a dozen "typical VFR waypoints" (churches, oddly shaped lakes etc) into the GPS as lat long once. After 25 mins of punching and twisting, all the while hooked up to the battery cart, I was convinced I'd never do that again. At least not while using panel mount GPS.