Oxygen is very heavy. That's the issue. Not carrying it (or carrying only the bit you need for the portion of your journey that is in space) makes a HUGE difference.
Exactly! Most of the mass of rocket launch into space is fuel/oxidizer (at liftoff, the Saturn V was ~95% fuel/oxidizer by mass). And if you're burning H2 as your fuel, you need ~8x that mass of O2. Being able to use oxygen from the air is a huge carrot that can justify quite a bit of extra equipment mass (particularly when you start talking reusable).
I've long thought that SCRAM jets would end up being the answer - when material technology caught up with the thermal requirements. But if this works with the current technology, that's great.
In the year 1800 (and for centuries before that), rapid long distance travel was around 6 mph using a sailing ship or horse drawn wagon. 100 years later, in 1900, that speed had increased to ~60 mph, using trains with steam locomotives. 100 years after that, by the year 2000, that speed was around 600 mph using jet aircraft. Is it that far-fetched that by the year 2100, that speed could be around 6,000 mph using sub-orbital transports?