It was perfectly safe to hand fly the aircraft even at Mach 2. There was at least one legendary captain who always believed in hand flying. The controls were not overly sensitive as the outer and middle elevons
The Captain I believe you are thinking of, not only hand flew the aircraft for the entire sector, but also ate his meal, and talked to the passenger who visited the flightdeck whilst he was flying it
He was not unique,and many would hand fly it during climb/accel and for the descent but most pilots would engage the Autopliot especially for supersonic cruise