I quite agree.
I had a very similar experience. I had, of course, read the Flight Training book section on stall recovery before I actually had to do it for real, and thought I had it firmly fixed that, if a wing dropped, you corrected it with opposite rudder.
However, when I did my first stall training (I had only done around 8 hours) and the left wing dropped, in the heat of the moment I inadvertantly gave it full left rudder, with very alarming results! Chart and kneeboard were floating in thin air around the cockpit, and the accelerometer registered -2g! Needless to say the instructor recovered us from the spin.
In short it scared the s**t out of me, and from that day to this, I STILL hate doing stalls.
I've had my PPL for just over two years now, but the need to do stall recovery doesn't go away. With each new type I have flown, stall recovery seems to be a standard part of type training.
So, be assured you are certainly not alone, and just stick with it.
Good luck!