I move my hand to the vicinity if we are getting close to the limit speed. Both speeding up and slowing down. Is this not sensible to prevent an overspeed or ending up below speed? I don't physically touch it, never had anyone say anything.
The correct move if you are concerned with the status of the flight is to bring it to the other crew member's attention - not sneakily move your hand about.
That way you may bring both of you back into shated situational awareness when their mind has drifted - for which they should thank you:
"Aprroaching flap overspeed there, John", or "Getting a bit slow for flap 2 there, John." , or
you may detect a subtle incapacitation which has gone unnoticed up until that stage.