Parabellum,
If one or more of your passengers died during your low level return to base as a direct consequence of oxygen deprivation earlier, and you had elected not to proceed to the nearest airport, you would need a very good lawyer to get you out of charges of gross negligence and involuntary manslaughter.
Passengers may seem "alright" after a rapid descent, but with 300 or so passengers on board, there will be some with heart ailments etc., who need to get on the ground fast. Why take the risk?