Regardless of which was there first (the airport or the housing), general aviation is, generally speaking, regulated more to protect non-participants than participants. Rule 5 provides for this, but repeatedly (Barton, Biggin Hill, Southampton, most recently) does not work because pilots do not comply with it. The Authority doesn't prosecute, though, so that bit of the Rule is derelict and needs attention, either by way of prosecutions to make people comply, or by an acceptance that an aircraft which has suffered a power failure has some sort of 'right' to come to earth on someone's home...
I don't fly single-engined aircraft any more, but respect the opinion of a close friend who refuses to operate an SEP aircraft into or out of a number of aerodromes, including Barton, which he views as simply unsafe and, in respect of Rule 5, impossible to operate to or from lawfully.