It is all tewwibly frustrating, isn't it? You will learn to 'put up with it' - and that it works both ways too. It may, of course, be that the Captain was trying to 'nudge' you as he/she may have noticed you were slow to back up the pointer? Who knows? It need not be the end of the world as we know it.
The other day my 'eager' hyper-active Jump-seat F/O deselected the ATIS frequency I was about to listen to and selected an Air Nostrum Malaga handling frequency so he could arrange his travel - all this as we passed Brest
and all without as much as a 'may I?". That must rate well up the
less that ideal atmosphere on the flight deck and is not good CRM
scale.