I think the reason is that it is very difficult to hand fly an aircraft at high altitude
Not true. I currently fly a heavy wide-body aircraft. In non-RVSM airspace, I've hand flown this aircraft (and several other heavies) at high altitude - up to 41000 feet without any difficulty what-so-ever - and held altitude, track, etc, to the same standard as the autopilot (Which may not necessarily being saying much)