I keep mine on. I leave it on my probably inflammable jacket issued to me by work. It saves trying to find it or worse, leaving it in the car.
I can understand that during the day on a small relatively quiet field that you don't need them. In fact, never wore one when I was in Oz. Not even wandering around aeroplanes at Sydney airport.
These days I wander around aeroplanes at night with vans and fuel trucks whizzing by. Since I am wearing black it's probably not a bad idea to keep the hi viz jacket on.