I had a similar issue where my sunglasses were lifting the headset seal so it was uncomfortable and the noise difference was noticeable. I went for a pair of Serengeti sunglasses with the S-flex arms. I can't remember the model name, but they're non-polarised and I've found them to be ideal.
The arm is flat and sits comfortably against the side of your head without adding pressure or affecting the headset seal. At the hinge the flat arm bends inwards to move the whole arm closer to your head.
Google "Serengeti S-Flex" for some close up pictures of the hinge.
-j34-