The more direct analogy is with mini-cab drivers - who do not need a special driving qualification.
Largely the issue is about protecting the commercial aviation companies. Most of the prosecutions come from AOC holders telling the CAA about people who are running illegal hire operations.
This is not to say it is a bad thing to protect the public in a specialised area where the majority of passengers would be unable to make an educated decision. It is likely is the Acme aircraft cab company set up business their accident rates would be higher than the present arrangements, commercial pressures would pretty much ensure that.