There are several other potential problems. Impedance and signal mismatch can do odd things to small solid state stuff. If the sound level drops a lot, or the sound quality goes for the chop, that's a good warning sign not to do that.
If you want to mix two different signals into a third device without such problems, all you need is a small active mixer device - effectively, a tiny stereo amplifier with two inputs matched to the "driver" devices, plenty of separation, and a stereo output at the right impedance for the following device.
You can probably find something suitable in one of the gadget shops.
I built one last year for a PPRuNer whose requirements were a little more specific, but I'm not going to volunteer to make any more - the bits cost about a quid, but the box to put it in, and the cables and plugs were absurdly expensive.