oldpax The main reason appears to be the usual one: Money!
In the USA, when Hub-and-Spoke ops were developing, transfer times were (are) often tight. So pax learnt that if they took bags with them in the cabin - they were more certain of making the connection. If the carriers insisted that cabin bag rules were enforced, then pax would go to another carrier that did not mind.
Also, on single sector trips, if carriers let pax flout the rules and bring more into the cabin - so that they do not have to check it in - then they get off the a/c faster and feel that they have enjoyed the purchasing experience even more. Stop the pax doing it and they will go to a carrier that does let them do it.
So: which carrier will enforce the rules first so as to ensure that they get a reputation for not letting you take your bag in the cabin?