Depends on what you mean by "ripped off".
To me it implies a lack of transparency of charges.
If the charges are published, that's OK. You can fly to Gatwick EGKK. 24hrs PPR, and about £500, but it's not a problem at all. You can call up Harrods Handling and they will give you the costs, so it's not a ripoff
I was at Luton (by car) last year, at a seminar. Two blokes flew in in a PA28, and discovered the £400 handling fee when they got there, and both were very obviously upset. They were probably victims of standard UK PPL training (zero operational knowledge) but they should have checked before flying.
Same with most cases of handling agents. They are normally far better run businesses than the airport itself which is often run by morons. You can call up the handler and get all the stuff from him. This is why a lot of higher-up GA pilots prefer handling; for the £100-200 you get a decent service (PPR etc all organised and Customs lubricated, etc) and if it is a one-off trip the cost is OK.
If you fly there regularly then you sort out some locally known "system" for avoiding the handler.
It is when you are trying to avoid handlers on a one-off basis that you sometimes get caught. There is a kind of Holy Grail in "tight" GA and it is called "avoiding handling". Every subterfuge is used. "Taxiing to the aeroclub" is a popular one. At Norwich it was either Anglia flight training (or something like that) or a particular hangar which you could just walk through. The locals know all this. In turn, the handlers are fighting their own battles, and some are crooked, and the further south you get from "white christian Europe" they more openly crooked people are, in general