Whilst I agree with Klauss' comment that hand flying is rarely done at altitude, I can't agree that it presents "not a problem at all".
Even for a pilot with high handling skills - (a descriptor that I think, if we were being honest with ourselves, very few of us, in these days of maximum use of automation, would lay claim to) - flying at high altitude, even if not in Alternate Law, can be a very mushy can of worms, even in smooth conditions.
In an area of severe turbulence, hand flying at high altitude is not something I'd like to have to do, particularly with such a narrow speed band as the unfortunate Air France crew presented with.