Yes that's correct.
However as you typically can't actually see whether you're within 500' of a person (people get up to all sorts of things in the "open countryside", eg a courting couple deliberately hiding under a bush) this is effectively a minimum 500' height over land.
Over the sea, more than 500' from any boats or swimmers or whatever, you can go lower. But whilst you can fly one of those <== around quite happily at 100', and I have done, I wouldn't recommend it if you're wearing wheels - what happens when the fan stops?