Actually they are extremely easy to fly with A/T off, at least the A321 I flew for seven years. Provided the thrust changes are not too brusque, the aircraft will maintain its path, while any speed deviation is quickly spotted with the speed trend vector: if the trend vector points forward you pull the thrust levers back a little and viceversa. You simply set a flightpath with the side-stick and then control speed with the thrust levers. Really a sort of point-and-forget aircraft.
Almost too easy to handfly, and consequentially a handful when you go back to direct law (or a pre-777 Boeing or MD) because you are no longer used to monitoring attitude (and trimming) any longer! Or at least that was my experience.