Exactly, the P-factor causes the aircraft to yaw to the left (of course having in mind clockwise-rotating propellers), which actually happens all the time during flight - takeoff, climb, cruise, descent and landing.
However during cruise the rudder is trimmed (via rudder trim or fixed trim tab) so force required on the rudder pedals is zero. The yaw which is produced by P-factor is actually variable with speed, since the effectiveness of rudder increases with speed - this basically means that with the same engine thrust and reducing speed, you would have to increase the opposite rudder to maintain zero side-slip angle.
That having been said, I wouldn't recommend doing powered stalls (not approach to stall) without an instructor on board, since the aircraft should not roll or yaw during stall as you say it does. Perhaps you could ask an instructor to demonstrate it to you correctly.
Here is another good discussion on how to maintain wings-level attitude during stall.