USN use 100% Oxygen from the get go, so they would normally wear the mask from start up. The (possibly) only fast jet pilots that don't wear a mask (during displays) are the Blue Angels who use a boom microphone.
Personally I would always wear one in the aircraft, apart from eating and drinking, as a bird in the face, rapid decompression, canopy failure or canopy jettison/MDC splatter can really ruin your day without one.