I seem to recall that the rudder deflections corresponded with pedal movement. Since the only system to back-drive the pedals is the autopilot, which was not engaged, I think it is likely that the 5 rudder deflections were a result of pilot input.
Even more conclusive (IMO) is that pedal inputs preceded rudder deflections.
Also looking at the pictures of the rudder fragments, I get the impression that the rudder structure failed first, followed by the actuator and hinge attachements. The reasoning being that the fractures across the rudder surfaces where caused by tensile loads rather than tearing. If the actuator attachments/brackets had failed first, there would be nothing to impart the magnitude and type of loads to cause the structural failures we see.
The fact that Airbus and FAA pilots could not perform partial rudder deflections (i.e. it was all or nothing) during recent tests is also significant.