However, when you look at the rules for operating Class G/VMC, the differences betwen IFR and VFR are very subtle - with the most dramatic difference probably being that quadrantals and flight levels are recommended VFR best practice vs. IFR requirements.
I would hardly call the minimum height rule a "subtle" difference