The Airbus normal fly-by-wire behaviour is pretty much like other types in control wheel steering: it wants to keep flying the last commanded attitude. The problem that I can see is that, with the A/P off the pilot with his hand off the side stick may easily let his attention be diverted to other tasks (as can typically occur with the A/P engaged). As the total energy state and flight path divergence may no longer be appreciated by that pilot his reaction time may be impaired because his CONTROL hand isn't guarding the @##&$#@-$ CONTROL stick.
How basic can you get, FFS?