Look at the number of prosecutions for low flying each year. Close to zilch.
CAA Prosecutions | About the CAA | CAA
How the hell does joe public know how high you are anyway?
Unless they have some sort of measuring device (unlikely).
I guess they could get the radar trace, but they would only do that if you either crashed/caused a crash/flew over some sort of gathering/Head of CAA's house at 50 feet.
I agree with Nigel, just carry on regardless!