So, Lord Lucan, what do you do? A passenger is legally required to do what the Captain says, as long as it is reasonable. Being asked to remove a badge, I would say is reasonable. If the passenger refuses, then the Captain has the right to assume that he will not obey other reasonable requests ("Please make sure your seatbelt is fastened sir", "Well why should I, you are tying me to my seat against my will..."), so for the safety of others he removes him. The removal of Mr Gilmore from the aircraft was nothing political or anti free speech, he just failed to respond appropriately to a reasonable request.
I must admit, that I agree that some of the security measures are not actually increasing the level of security (some quite the opposite) and that we have to be careful of some of the schemes people have come up with. Also, I do agree with free speech, but we have to put things in perspective. Mr Gilmore is trying to make a point, when he has made it, there is no sense in pushing it to the limit to drive it home - the point is made.
As for taking BA to court - what a gigantic waste of the public's money. Far better to spend it on a hospital or school. Of course an airline can refuse carriage, just like a pub can refuse to serve you.