They are symbols which resemble the 'bugs' on the associated instrument. On the original electromechanical-fitted 737-300 (and some later -200s) the HSI had a heading bug like a triangle with a slot in it. It also had a course line which had a top like a dagger. The ASI had a bug like the nib of a fountain pen (remember them?). The symbols on the MCP knobs match those, in an attempt to remind the pilot which parameter they are trying to control. (The bugs on the newer EFIS-equipped machines may not be identical to those on the MCP. An interesting historical hang-over.)
Despite that, and the fact that they are shaped differently and feel different when they click around, I and others have selected the wrong knob and then watched the wrong instrument, confused, as the aircraft does not respond as expected. Like the person who thinks that the lift will arrive sooner if he pushes the call button multiple times, I and others have then moved the offending knob even further, in an attempt to get the stupid aircraft to behave!
I remember one particular night accelerating towards Mount Teide at TFS instead of turning right towards the ILS LOC. How we laughed....
As far as the symbols on the F/O's and Capt's CRS knobs being opposite to each other is concerned, I suspect it is just the way they have been mounted. I don't know of any reason why it would matter.