While airlines will never be able to get it perfect every time I still think that more can be done. The revenue management systems that manage overbooking are prioritised to make money, not to minimise passenger inconvenience. If they aimed for a 99% load factor, rather than 100% (for example) there would be fewer gate negotiations.
They should have a very good idea of what flights are going to be in danger of spilling passengers. Making offers 24/12/8 hours out, with small incentives, could shift passengers to less full flights. There actually are some customers that would like to change but are put off by the charges for doing so.
The gate experience is the worst part of the journey, IMHO, some clear thinking could improve this immensely.