You're forgetting secondary emissions
NoFlyZone, you're forgetting secondary emissions.
Just because something is "not a transmitter" doesn't mean that it emits no RF. That's why using your laptop is fine, but using an external mouse is banned -- because the mouse wire is not shielded, and emits a fair bit of RF.
Your VHF transceiver has an intermediate frequency oscillator. Depending upon the build quality, whether the case is metallic or plastic, and a number of other factors, it, when operating in RX mode with TX locked out, still emits some RF at the intermediate frequency. That's why the use of FM and TV radio receivers is banned on aircraft.
If the IF of your transceiver is the same as the IF of the aircraft's comm radios (which it is likely to be), then there's the very real possibility of interference.
Yes, the chance is slight, but, at the end of the day, the process by which "everything not specifically approved is banned." is intentionally conservative, which is pretty much the way you, as an occupant of the airplane, want it to be, no?