The A320 is presumably always under computer control, just the extent of it modulated in case of glitches to alternate or direct. That the FAC would likewise have residual action seems unsurprising. How those specific demarcation lines are laid out or understood by the crew is less clear.
It does seem that a very experienced Captain and a mature and dedicated FO felt they were at the controls of an unresponsive or defective aircraft.
They may have been mistaken, but how that mistake came about needs to be made clear, if only to prevent recurrences.