For pilots on duty travel heading back home they will often remove wings etc' or put a sweater on so that they can have a drink. The general rule is no alcohol can be purchased or consumed while there is anything that identifies you as being part of the airline. The other side to this though is that if you do have any part of the uniform on then you have to wear all of it. That is to say, you can't just go half and half (say wings on and no epaulettes).
It is also a case of wanting to be somewhat invisible as 18-wheeler points out. I know when I am on duty travel heading home it is not uncommon for the passenger next to me to ask all the usual questions about flying if I am in uniform. I have no problem with this and I will always oblige, but sometimes if I just want to sleep or relax it is easier to remove all evidence. I have to say I feel a bit uncomfortable when people start staring at the uniform but that's just me.