It used to be "not at-all", nowadays it's basically rule 5, the same as everybody else. That is 1000ft above the highest point and always able to glide clear. + 500 ft separation except when taking off or landing.
Of course, what comprises a built up area has never been adequately determined.
If he's being a prat and a nuisance, take his registration, look him up on G-INFO, write him a letter telling him that he's done it enough times, and if he does it again, you're reporting him to the CAA. It should concentrate the mind, and you may well save him from eventually having a totally unnecessary accident as well.
G