I usually remove mine as soon as I leave the terminal. That said, such a process has caused me to lose them so I sometimes wait till I get home to remove as well. I place them in the same spot along with the wings and other crap so that I know where everything is before the next trip.
I never wear them into resturaunts, malls, shopping, etc. Work is work. Why would you want people in a mall or eatery starring at you anyway?