My main reasons for being +500 ft AGL (most of the time) include safety (more options if the donkey stops), compliance Rule 5 (at all times), reduced noise pollution (less complaints about GA.
I fly low (below 1,000 ft AGL) on occasions that I am required to (eg one section of London Heliroutes), I have to to remain VMC and legal (airspace / Rule 5), to land & take-off (often off-airfield) and sometimes because I can and it is fun (down the centre of the Great Glen, sections are quite legal at low height).
"Justifying" all GA flight below 500 ft on safety grounds is not possible. Much GA flight is not at all necessary, and as there is inherent danger at any height then I could not justify any of my flying at all.
Your last post suggests to me that (a) having other low-level traffic is actually useful (though this little B**d doesn't have a gun) (b) that you brief for the flight, which would include the very obvious point that other aircraft may be using that space, they will be below 250knots, you will be difficult for them to see (I think that's the point of your paint echeme) so you have the duty to avoid, they may not be carrying a transponder (not everyone does, though mine is Mode S) and given what they are made of they might not have a big radar signature so you will need a very good look-out (which you confirm you have).
There are many commercial operations that will have people operating in that airspace. There are numerous landing sites (unmarked) - particularly for RW (picking up from a large garden, for example?). There is noting illegal or requiring authorisation to be below 500 ft AGL, so long as you are 500ft clear of any person, vessel, vehicle or structure; if landing / taking off then not even that.
You can't have everything. If you want to use general airspace to make things more realistic, then you are not going to be alone. And incidently, I am pretty sure that RAF uses my home as a practice bomb target and I think that's just absolutely fine. However, the helicopter pad East of the house is there for a reason.
Oh! and the times I have had (for example, my last one) "Pop-up traffic, left to right, 2 miles, Level, high speed" messages from ATC I have been slightly +1,000 ft AGL (a nice place for RW) so I don't personally feel any safer from potential conflict with military if I agree to your request.