Unlikely, I think. I've had full rudder in at low speeds in the simulator with engine failures and never felt that the roll authority was limited. I would also expect the spoilers to always be able to overcome the rolling moment of the rudder.
I can't conceive of a rudder hard-over event occurring on the A320 series - it would require, essentially, a fault in four independent computers (2 x 2 command and monitoring channels) and a failure of G+B or G+Y hydraulics.