I should have thought the 500ft rule makes it easy to make a decision, especially when there are passengers aboard or waiting to be collected.
"Sorry, I can't go on, it's illegal!" The decision is made for you by the legislation, you can blame the regulators - if you want.
Getting below 500ft to, say, 400ft, means you're taking the decision-making onto yourself and now you've got nothing to hang your hat on, as it were, except your judgement.
Given that there is evidence of flawed judgement to begin with by breaking the 500ft rule in the first place, I'm of the view that you're now fully established, so to speak, on a slippery slope.
And they only go downhil.