I saw the genesis of this problem first hand but I didn’t realize it at the time, nor could I have since I wasn’t a pilot.
Basically, airline owners and investors started asking circa 1980: “what do I get for all this money I am spending on new aircraft with automation, INS, Omega, glass cockpits, etc., etc.?
The answer, for a while, was increased operating efficiencies, fuel, maintenance, etc.
But ultimately the answer had to be smart aircraft = cheap (dumb, inexperienced) pilots.
We are now experiencing the costs associated with that philosophy - Colgan perhaps was the first wake up call.