I think you were wrongly refused that refund by BA.
If you go to
AskBA and search for "1647", you will get Answer ID 1647 which says:-
...
4) If you have a BA ticket which you no longer wish to travel on, for your own personal reasons. In this situation, please be aware that your ticket might not be fully refundable and you may only receive a partial refund or a refund of just the taxes/fees/charges that were applied to your ticket.
...
Please note. Each fare type has conditions attached which may include it being restricted, non-upgradeable or non-refundable (taxes paid are refundable). Generally the lower the fare the more restrictions there will be, however as we sell numerous different fare types and combinations, we are unable to advise on this web site just how much it might cost to change, upgrade or cancel your specific travel plans.
If this was a published fare that you were travelling on, bought from BA, I would get back to them and escalate the call to a supervisor if necessary.
If this was an unpublished/nett/consolidator type fare, you probably need to get back to the travel agent who sold you the ticket.