Actually they aren't oversized - they are sized for the design case, which is usually the VMC condition.
The reason crews should not be cavalier about rudder use has nothing to do with the sizing of the rudder for the VMC case; it's all to do with the structural design requirements and the defined design manoeuvres, which do not include aggressive use of the rudder. So any such aggressive use will exceed the design loads in all likelihood, and if persisted with then structural failure is likely.
Thats even the case where the rudder design has nothing to do with VMC, such as on a single engine aircraft.