I'll need to look it up, but there is a mechanical linkage of the pedals to the rudder as per FCOM. Should work both ways, but perhaps introducing some freeplay?
Would Yaw Damper function or just kinematic / aero lag (WTH, on a powered rudder, why?) explain a bit of what you are seeing? I can't really read it at first glance. I don't think there is any FBW filtering / gearing for the rudder at all, save for that Yaw Damer function (which is not fed back to the pedals - so much for a mechanical linkage).
Once I believed the link actually only goes as far as a hydraulic clutch, into the tail assembly, where it interfaces with a closed loop internal servo-booster of the rudder itself.
Otherwise the YD on the bus is underexplained in the FCOM and that is an understatement. But hey what, they need to say steady climb is a constant 1 g path for their audience.