Martin, if you think that there is some element of a scam and time permits then, ask the CAA for a justification.
If the 'justification' looks reasonably watertight, then you really have little option but to pay it.
If the 'justification' appears high on the oxometer scale accompanied by a strong whiff of BS, you could pay under protest, and then test the matter in the Small Claims Court. That could be interesting !