I've read that the most common cause of pilot incapacitation is.... choking.
I can see that conceivably you could drop your glasses behind the seat or out the door depending on the aircraft type. It's a risk.... but how much of a risk? There's lots of other things you could do too, but hard to draw a line eh.