This is an interesting take on the security (not safety) aspects of hypersonic and sub-orbital air transport:
Why we're not going to see sub-orbital airliners - Charlie's Diary
In short, the author asserts that it will not happen since the risk of misuse is to big: a multi-Mach, multi-ton piece of titanium and plastic makes a good kinetic energy projectile, and regularly having those on trajectories pointing directly at densely populated areas would not be a Good Idea - mainly because the warning time between detection of malicious intent and impact is too short.