Tha medical departments in the national CAAs are the most powerful factions within these organisations.
Since I started flying in 2001 I have been watching the various (mostly totally unsuccessful) attempts to deregulate various bits of private flying, and the medical stuff has stubbornly stayed put.
Some years ago the CAA relaxed the initial audiogram limits to the then existing renewal limits, but a pilot with one perfect ear and one bad one could still never do a JAA IR. On a renewal he could keep an IR... Current EASA proposals seem to overlook this completely and keep the old JAA system.
I am in two minds regarding doing away with medicals totally. Even a blind person can see we (in the West) are on the verge of an obesity epidemic, which will translate in due course to massive heart disease, diabetes, and other general health issues. I tend to concur with the view of a CAA AME I know who thinks the Class 1/2 medicals have little value in picking up medical conditions which would result in in-flight incapacitation but the mere existence of the medical causes pilots to look after themselves a little better than the average population (which is heading steadily downhill) and that is a good thing.
If somebody never carried passengers, I would have no issues with no medical.