The first group are generally employees and have come under the dominion of the bean counters.
The second group generally work as individual practitioners or in partnerships and has the power to set their T's & C's so that they can have a reasonable lifestyle.
Both groups require some intelligence to do the job, but the second group requires credentials that take several years to acquire and restricts entrants.
Now line pilots do at least usually have a union that prevents mass replacement of the older, more expensive pilots with guys fresh out of school. This is good as long as your employer remains in business.
The whiz kids picked up a while ago that programming has become an entry level occupation and gravitated to the financial industry, where so far the Indians and Chinese have not (yet) taken over.
Eventually the accident rate will slide back up and beatings will increase until morale improves while the authorities promulgate ever thicker SOPs
Eventually there will be nobody up front and all flights will be run out of Bangalore by floors full of dispatchers communicating via uplinks
