You are absolutely correct when you note VFR/IFR is based purely upon weather conditions alone. The visibility minimum only applies to the distance one can see "If" there was something to see.....such as lights. horizon, terrain. Anyone that has flown offshore at night knows what the difference is.
I have always maintained that in reality, if one must rely solely upon the instruments to control the aircraft you are no longer "VFR" despite the regulation saying you are.
The issue in my view is the regulation does not adequately meet the Kool-Aid Acid test of reality....and thus sets up a very dangerous situation where un-qualified, non-capable ( I hate to use the word "incompetent"...) pilots can suddenly find themselves in grave trouble.