Bear in mind that 500' isn't an easy height to guesstimate from ground level...+-50' or even a little more wont make much of a difference to any casual onlookers.
Without proper low flying tuition, flying below 500' probably isn't a good idea...even flying at 500' requires a lot more anticipation for the simplest of things - like climbing to avoid ridges. Plus you can't rely on your altimeter to give you an accurate altitude AGL because of the changing heights of ground AMSL, which means you need to be able to visually judge your height.
Generally with a good lookout fast and pointy things aren't impossible to spot...another good reason for sticking to 500' - their lower limit is 250' meaning you decrease your risk of meeting one coming the wrong way if you remain at that height (that's not to say all fast jets will always be at 250' in the low flying network!). If you're really worried stay away from the FJ hotspots - like the Welsh valleys, the Lake District and so on - you can find them on the internet.