Genghis,
I think I get your drift, but I'd suggest that the key is VMC/IMC not VFR/IFR, in the circumstance that you infer.
I had a co pilot who was the Base Manager (and from the USA) who logged all flights on an IFR flight plan, as IFR, regardless that he was not on the controls most of the time
My interpretation has always been the caveat "sole reference to instruments", and if the horizon is fully visible and provides a visual reference, how can you claim IFR?