In the UK it is perfectly legal to fly as low as you wish as long as you maintain the required 500ft separation from any "person, vessel, vehicle or structure".
I sometimes do low flying in Brecon Beacons, but I use tricks which help me judging my height with great precision (using my shadow on the ground, etc.) and I still add a huge safety margin to it. I think the question should not be whether you do it or not, but how serious you are about doing it safely.