Commercial air transport safety revolves around statistical probability of various kinds of event.
The reason why it's OK to continue flight at normal cruise levels on one pack is that following a pack failure, a failure of the other pack is considered highly improbable, and therefore no special mitigation is required.
Put another way, it is considered that the probability of a poor outcome from a single pack cruise at normal cruise levels is as low as other acceptable risks.
If you genuinely
would not like to be in an aircraft operating on one pack at FL400
, then you need to start worrying about lots of other hazards, too, all at about the same level of probability.
As I said earlier, it's best simply to do what the books say, rather than attempting to second-guess them.