Swapping glasses in flight won't be fun, with a headset on. There are times during a flight when you start with sunglasses, then don't want them, then want them back. Each time, it will be a dual action for you, one pair off, another pair on. Interim moment each time when you are without correction.
I wear contact lenses with plain sunglasses. It works for me.