"I do accept that whatever the PIC says goes - so no worries - but am curious as to WHY someone would not allow the use of a handheld GPS receiver in the back. It is a after all a RECEIVER - or does it actually transmit anything that could interfere with instrumentation?"
In theory: All radio receivers also emit a signal from the local oscillator. The better/more expensive ones emit less of it. The GPS signal is especially weak and sensitive so it is possible to be interfered with. I have noticed that running several GPS receivers in close proximity to each other causes reduced signal quality and sometimes loss of signal on some of them.
In practice: The aircraft equipment is much too far away from your receiver for this to be a problem. It also has nice big antennae and high quality amplifiers. And, the pilot doesn't know about this. Almost all pilots wouldn't mind you using a portable GPS so in reality this is probably just CC being jobsworths.