I don't think it is a case of forcing GA out, more that the bigger airports do not want or warrant the infrastructure to collect the fees and deal with the small aircraft as it is not profitable. The handling agents exist to support the larger GA operators and as such they have the ability to process these costs. They have overheads and as such expect to cover them.
In the UK airports are self funding and have to make a profit for the shareholders. Someone has to pay for this and that is the customers or the airport of which us as GA vsitors are one. The numbers of small aircraft that visit big airports are miniscule.
Personally I don't see the issue with paying for a handling agent as long as you get the service you expect. That includes things like arranging Taxi's, weather briefing, booking in out etc. Most places I have been to do this very well.