Tosser or not, unfortunately, this is just how it is in the big wide world of Class G.... One can't assume that a radar service will pass information on all other traffic. My personal concern around the edge of the airspace is powered aircraft operating inside Luton's airspace without talking or squawking, sometimes whilst towing another aircraft behind..... Radar seldom see them, or if they do, they don't tell.
In your Airprox (you have filed, haven't you?) it was possibly the case that a pilot not 100% sure of his position may have been looking from ground to map when he whizzed past your window.