A good job your passenger didn't try to smoke with Oxygen!
Smoking, like altitude sickness, is one of those things where people differ a lot in how they cope with it. Occasionally you'll get a 30 year old smoker whose lungs have already started to fall apart, and a 90-something year old who seems impervious.
But no mystery regarding passenger v. pilots - surely. It doesn't matter if passengers are none-too-sharp provided they don't suffer lasting harm. If you're a pilot, you need to stay alert. And given that they need to set the limits such that almost all pilots will stay alert at 12,500 feet there should be no surprise if quite a lot of people can cope with going considerably higher.