Passengers who get stuck in traffic, or are unable to travel due to injury/bereavement/illness etc have come to expect the right to a refund, or at least to be able to transfer to a different flight (varies with class of ticket). So the airline don't know whether they actually will have a full load of fare-paying passengers until just before closing the gate. The continual competitive pressure to reduce prices means hat flights are costed to break even on ever-higher levels of occupancy, so a few no-shows who then use their tickets on a later flight can result in a loss-making flight.
The airlines (like hotels) have tended to address this by deliberately over-selling flights on the assumption that some passengers won't turn up. Most of the time they get their sums right and it works, but sometimes it doesn't. You then have multiple people who all have what they see as a legitimate right to a seat. The airlines have a "priority" system which allows them to decide whose right is superior - types of ticket, frequent-flyer status, check-in times, lone or group bookings etc can all be a part of this, but if enopugh passengers have a "guaranteed reserved seat" or similar then they will end up in the auction to sell the deferment to the lowest bidder. If they get no bidders then (in many countries) there are legal provisions for compelling people to stand aside before boarding. If they do this before boarding then people will generally grumble, but comply. If they try to do it after boarding they have no legal basis and they find themselves splashed across social media and facing huge lawsuits.
There are alternatives:
1. They could put the prices up such that a flight with a 10% no-show still makes enough money to not need to fill the seats, and then over-selling would be unjustifiable and could be outlawed.
2. They could make all tickets non-refundable and non-transferable - you bought that seat and if you don't get to the airport on time then that's just tough. You could then (also) make over-selling illegal. It would mean the end of "flexible tickets" as we currently know them, and I could see a procession of injured/bereaved "victims" replacing the bumped passengers on social media, but as a system it would work.
3. You could enforce a strict legal "first come gets on the plane" physical check-in system, but that might be seen as discriminatory agianst those who use public transport that went on strike, or those who got stuck a mile away for an hour because of a crash on the motorway or a group of plainly stupid protestors.
Of course none of this applies to the UA incident at Chicago. That was just the airline forgetting that it had no legal right to deman boarded passengers give up their seats to dead-heading aircrew. That was just a mistake, and I suspect it's one that will cost them dearly.
So they just to to learn to make fewer mistakes...
PDR