My TR instructor always used to say that Mr Boeing put those armrests in for a reason, so use them!
Personally I keep them down and find that when flying any kind of approach, it can be quite handy to keep your elbows locked in position on the armrest and move the yoke with your fingers. It helps maintain a reference attitude in terms of remembering where your arms actually are... You can fly a raw data ILS with much greater precision this way.