It's the 1,000 foot (rather than the 500 foot) rule that catches most (helicopter pilots) out and puts them in breach of the law.
You need a written permission from the CAA to disregard the 1,000 foot rule to land at a site that's in a congested area, other than a government or licensed aerodrome. Many don't bother, especially as it costs money.