Wearing your uniform when not on duty is akin to a fireman,policeman,doctor doing the same.These people normally dress when they are on duty, not on the way to/from work. For those of us without a locker/cupboard it is not too difficult to have our uniform tucked in your pocket that can be promptly displayed when necessary.
My philosophy is to display my uniform in the environment that I am paid for, not anywhere else.
If you were paid to where a clown suit at work, would you wear it anywhere else?