Provided hand is coupled to FCU, not to stick. That needs not be everyone's (or even majority's) definition of manual.
I don't understand your comment. Its superfluous.
If your hand is "coupled" to the stick and your making inputs, you're going to disconnect the autopilot.
You cannot make stick inputs with the AP on in the Airbus - period.