In very windy conditions you may well have to hold the sidestick against a left or right stop for a couple of seconds before anything seems to happen. However, remember that there are always two of you flying an Airbus -
even with the A/P out - you and the flight control system. Even in manual; the aircraft will already be recovering the roll - maybe already using max control deflection - so your input may have no extra effect for a while until the gust subsides. The aircraft will be recovering as 'fast' as it can in the circumstances - but not overstressing the airframe in the process.
It is a very different technique to using the traditional mechanical yoke, but I find that once you understand the concept that in manual control you are effectively guiding an autopilot all the time, it becomes easier to get the hang of.
I think of it like this: The autopilot disconnect in an Airbus changes where the autopilot gets its navigation commands from. With 'Autopilot in' the navigation commands come from the FMGS/FCU. With 'Autopilot out' the navigation commands come from you and your sidestick, but
the flight control system is still helping to fly the plane.
Took me a while to work this out - and of course; no-one ever tells you.