My answer is no, we don't have to comply with the 500 foot rule when landing at a private site in the UK, as long as we are flying in accordance with 'normal aviation practice'.
If the CAA had meant to exclude private sites/strips, they would have said 'at a goverment aerodrome'.
What constitutes 'normal aviation practice'? Well, I interpret that to mean not directly overflying a person animal vehicle or structure on my approach....I can get closer than 500 ft during the arrival/departure, but should be prepared to vary my approach path where possible to minimise the danger/inconvenience to those/objects on the ground. And I can land closer than 500 ft to my neighbour's house.
But it's a good question - is there a better definition of 'normal aviation practice' in this context?