Hi Beechnut:
Personally, I don't think that you have justify wearing a blue woolly pully while flying. I myself have a black one and a green one (courtesy of HM the Queen), and indeed they are reasonably warm, not too bulky, and fairly smart looking. But like you, I never wear them with miltary rank, badges or other insignia.