If you are really concerned about what people think of you, there are lots of nomex products that you can buy that don't look like a one piece flight suit.
Have a look at Drifire, they make a bunch of nomex clothing, including a two piece flight suit. If you get just the pants, they kinda look like khaki cargo pants.
FireFighter Protective Clothing, Fire Rated Clothing, FR Clothing | DRIFIRE
DRIFIRE All-Weather 2-piece Flight Suit, Military Flight Suits | DRIFIRE
Also, have a look at companies that make apparel for racecar drivers, there are lots of Nomex underwear products out there.
Under Armour makes some fire resistant clothing as well, it isn't nomex but modacrylic, which is also flame resistant.
Men's Fire Retardant MOD Longsleeve T | 1216034 | Under Armour US