I have never known an Airbus FBW autopilot to suddenly drop out by itself*
It's very much possible. If you experience any one of the twenty two failures given in FCOM the aircraft will go in alternare law tripping the autopilot. But since it will maintain it's flight path doesn't require frenzied handling of the side stick. Even when the airbus is flown manually the side stick is used to command change in flight path and then left alone. So with AP on it doesn't make any sense to keep holding the side stick.