I've been involved with building and flying model aircraft since I was a small lad. All the clubs that I have been involved with took safety very seriously indeed. To fly at a club designated flying site we all have to be a member of a prefessional body such as the BMFA, LMA, SAA etc, they provide safety training schemes and 'Pilot' proficiency tests, also all members are fully insured against accidents. Unfortunatly like most things in life you get rogue elements who just buy a ready made model, try and fly it without proper training or supervision and are probably uninsured. These people dont care where they operate and will happily try and fly next to an airport or built up area and give the hobby a bad name. At the site I currently use (near Banchory in Scotland, well outside Aberdeen's controlled airspace) we do occasionaly see low flying fast jet traffic flying near by, sadly at the speeds they travel at they have passed before you even hear or see them so it would be impossible to try and avoid. Having said that we dont fly large aircraft which means that we dont fly very high (less than about 150 feet on average) as you simply would lose sight and be unable to control them at greater heights.
Some probably do think its a nerdy hobby, each to their own I say, pretty harmless and far worse things we could do! Quite a few professional Pilots that I know also take part in this activity and one or two ex RAF chaps at our club (one is even ex CFS) so they tend to bring a good safety culture to the hobby. It's not as easy as some may think, it does require teaching from somebody competent or you can gaurantee a black bin bag full of broken model by the end of the day!
In summary: most model flyers are pretty sensible and operate well away from full size aircraft. Responsible, designated clubs should notify any nearby airfields of the activity, I'm pretty sure that our lot have told the RAF so they can mark it on the charts. Sadly the minority make it worse for those of us enthusiasts who just like to play with aeroplanes whether big or small.