I've got contacts and usually a pair of sunglasses on when doing aeros. Biggest problem is the headset falling off under sustained +g, when I'm looking upwards to mee the horizon (in a loop for instance). Putting the headband of the headset as far forward on the top of my head as I can without breaking the seal helps a lot, and indeed the headset keeps the glasses in place.
My biggest problem is jamming the required set of spare glasses into the map pocket so that they don't fall out under negative g.