You are right with your synopsis of sidestick priorities and logic.
However all inputs are fed into the FBW computers (ELAC & SEC) They operate within the parameters of a virtual flight envelope. So regardless of sidestick input, the aircraft will only manoeuvre within those parameters.
Only if the ELACs and SECs are switched off, then you enter what is called direct law and the control surfaces follow sidestick input.