Had you had the opportunity to repeat the approach with this problem a few times you would have become quite adept , I am quite sure . Flying is not natural nor intuitive for most humans - it needs training and practice.
No-one who flew aircraft 30 years ago would ever take an autopilot out without anticipating an out of trim condition. Autopilots would frequently drop out of their own accord due to turbulence and an inability to autotrim quickly enough .
I have checked on the A330 and a stabiliser jam is always assumed to lead to alternate law and hence ECAM warnings and procedures . One would hope that on the A320 the warnings would come with a severe out of trim condition and the procedure with no warnings would apply in a mild out of trim condition , although I appreciate that this does not fit with your experience .
A stabiliser jam is assumed to be recoverable although with restricted pitch authority.
The worst case scenario that you describe is theoretically possible - but there is no recorded case of it ever happening , which says a lot for Airbus, professional pilots, luck and statistics.