As far as I know there is no restriction on hand-flying in RVSM airspace; the restrictions are down to the equipment on board, not whether it's used or not.
I occasionally fly the A320 up to cruise and for a short time afterward with flight directors off. It's a very different beast compared to low level flight and very sensitive in pitch. You'll find yourself flying to about an 1/8th of a degree to maintain level flight.