Apologies if this doesn't apply in the UK, but the low flying regs in Australia have a clause saying something along the lines of, you're not illegally low flying if it's because of the inevitable stress of weather or compliance with an air traffic control clearance.
Our company have some low flying approvals but we don't do it willy nilly, only if it's required and doing things approved on the AOC. However, most of our flying around the local control zone is at 500 ft on QNH. That's usually the clearance, but I've been instructed sometimes to 'commence descent early to the pad' for example, and as long as I'm satisfied it's not a safety hazard, I'm happy to comply.
Do you guys not have a similar rule?