There are at least 2 issues here: Will a properly-operating handheld GPS interfere with the operation of the aircraft; and is such operation permitted by the operator?
I fly GPS-equipped 747-400s for a living, and also have a GPS-III+ that I use [mostly] for sailing. I've brought the GPS with me on operating flights and have found:
(1) the GPS won't lock on to any satellites in the cockpit (probably due to electrically heated windows and other interference);
(2) the GPS WILL lock on and track near any of the upper-deck windows;
(3) I have not seen any evidence of interference with aircraft systems.
As a pilot, I would NOT at all appreciate a passenger turning on an "unapproved" device without informing/asking the Crew first. OTOH, I would be inclined to allow such operation once I knew where to look if the unlikely interference bug bit...