The lungs would not be either deflated or collapsed. In a rapid decompression, they would, should you attempt to hold your breath, come close to exploding. Air would be forced out of your body. You would have no air.
Further, at low pressures the partial pressure of oxygen and nitrogen ensure that it is far more difficult to absorb oxygen than at lower altitudes, so although the partial volume of oxygen at altitude is the same at lower levels, approx 21%, you would absorb far less than that.
Another problem is with the "gasp reflex". In an explosive decompression, the temperature drops very rapidly. Your physiology with rapid chilling ensures that the body's first reflex is to gasp. Unfortunately, this is all wrong. You then immediately ensure that you can't hold your breath. There is a similar problem in immersion. If your aircraft hits the water, the cold water hits you and your body helps not a lot by immediately gasping, sucking in a load of water. Therefore, as you say, under non-stress conditions, although you can hold your breath for a minute or more, it is very very easy to drown in only a few seconds.