Modern jets can be flown by the tips of your fingers. Resting your arms helps keep fingertip sensitivity up. (Buls--t) I just can't do it without them!
Some 3rd generation jets like B727 with quite swept wings used to take almost full deflection to get them rolling. Guess back then you would have found the arm rest a bit in the way...