SS
In reality it isn't a problem and I don't know of anyone who operates modern Permit machines that treats the built up areas any different to a CofA machine. as far as I know, legally, it doesn't apply during the normal take-off and landing moments anyway.
Bizarre that in a 90kt Cirrus powered Auster you could fly over a village at 5000' (if it gets that high...) but in a 150+kt Lycoming powered RV you can't!
I presume the rule dates back to the days of Those Magnificant Men in Their Flying Machines and VW powered orange boxes.
Times have changed, permit aircraft performance has advanced beyond most certificated machines, the rules have stayed as they are.