Among other things the Rules of the Air Regulations 1991 state:-
(2) An aircraft shall not fly, take off or land within the aerodrome traffic zone of an aerodrome to which this paragraph applies unless the commander of the aircraft has obtained the permission of the air traffic control unit at the aerodrome or, where there is no air traffic control unit, has obtained from the aerodrome flight information unit at that aerodrome information to enable the flight within the zone to be conducted with safety or, where there is no air traffic control unit nor aerodrome flight information unit, has obtained information from the air/ground radio station at that aerodrome to enable the flight to be conducted with safety.
So, if it is within the published hours of operation and there is radio, you have to speak to them, whether they are a/g or AFIS.
So why pick on FISO's? In any walk of life there are people, who try to overstep their authority (traffic wardens, event stewards, local councillors.....) and the a/g operator is every bit as capable of doing it as the AFISO. Know the rules and object if they are being broken.
At least an AFISO has received formal training and has a license that can be taken away if he misbehaves. If you feel he is operating outside his authority, speak to the CAA and they may have a word in his ear.
Mike