I'm not a pilot, but have done a lot of flying as a 'free autopilot' in light aircraft when the CPL or ferrying Test Pilot was bored or giving me the chance, + a lot of 'simulator' time in varying degrees of realism, everything from multi-million £ jobs to PC games; my instinct with the light / FLIR would certainly be push forward, go down...
Incidentally, while not re. displays, this is a similar snag; I knew a very experienced Harrier Test Pilot ( not J.F. ) who for some reason had to test microlights of the 'hang glider with a pusher engine' variety; he remarked it went against the grain to push the wing control bar forward to go up !