Imagine the scenario: three or four IMC-rated pilots go flying VFR in something like a Piper Cub. (VFR, of course, because the aircraft is not equipped for IFR). The viz is 2000m, but at the 80mph at which these aircraft fly, that is plenty of time to see and avoid each other, as well as obstacles on the ground.
Then along comes Mr Twin in his nice snazzy 200kt aircraft, also flying VFR. Never mind how he's going to avoid the TV masts (he can probably fly high enough that it doesn't matter if he never sees them) - there's no way he could see and avoid the Cubs, nor they him, in that kind of viz and at that kind of speed.
Of course, in the UK, Mr Twin could just say that he is IFR, and suddenly it's all legal, with exactly the same problem. But these rules are ICAO rules which apply across the world - and in most of the UK it's very unusual to spend more than a minute or two in Class G airspace if you're IFR. Mr Twin would depart from his airfield IFR, and be in controlled airspace, out of the way of the Cubs, whilst still in the vicinity of the airport. Which is why, I think, the rules are as they are.
FFF
------------