The issue is that even the departure tax isn't a tax payable by the passenger, but by the airline. Accordingly, it's an airline cost, like fuel (if you don't fly, the aircraft will use a little less fuel - but you don't get a refund - and they are a coffee or two better off , again, no refund).
UK Customs and Excise think the airline has a 'moral' duty to return the tax, which basically means there isn't a legal obligation to return it, or they would have said so (asked someone there once!)
Under UK consumer protection legislation at least, it is likely that if the airline is adding charges for things such as tax, or airport charges, and stating them as such, the airline would then have to return them to the passenger. (I'm not aware of this ever having been decided either way, but could be wrong). However, the airline is perfectly entitled to charge a handling fee, particularly as a refund will involve a cost to the airline - time and transactional charges - and while it might be frowned on were that handling fee completely out of the ballpark of what is reasonable, it doesn't sound like many of them are.
The airline can't mislead you about having such a charge though - if it says, for example, that a ticket is refundable, in its terms of carriage, or if the travel agent tells you that on the airline's behalf, it needs to be made clear at that time that it's not wholly refundable.
Ramble ramble - sorry, but I have been asked about this before!