I should think that the problem is that fellow passengers could be alarmed by seeing people not wearing airline uniform (possible hijackers) entering flightdeck during flight. Could these fears be allayed by an appropriate PA message?
Speaking as a regular PAX and PPL holder I am more alarmed by the amount of times I watch the CC go into the flight deck on the short trips I normally take. Over the space of an hour on one flight I recall the cockpit door was rarely shut with a CC normally perched in the doorway, chatting to the pilots. I was concerned about the distractions being caused to the Flight Crew as well as the waste of a good security door.
I had a number of great jumpseat rides before 9/11 and now realise that those days are long gone (or at least for a few years yet anyway).
I personally wouldn't be worried about a 'pax' lookalike entering a flight deck, as long as the person entered the FD in a controlled manner.....
IMHO anyway....