But given that you cannot see the engines from the A320 flight deck, and that your eyes are on the E/WD monitoring the engine start-up, how are you going to know there are ground crew wandering in front of the engines?
No captain, including the senior TREs that I have flown with in my airline, keep their hands on the start switch during the start sequence nor have ever mentioned it. There is nothing in our SOPs about doing it either.
Having your hand on the master switch may cause you to make an instinctive reaction to shut the engine down when it would do no good, and as others have said, would interrupt the FADEC protections. Not touching the switch might just mean you take time to think about it for a second rather than instinctively yanking the switch off.
If a call comes from the ground that someone is wandering perilously close to an engine, then you could always move your hand back to the switch again... and if they're fatally close, then given the time it would take the engine to spool down, it would probably make no difference whether your hand is on the switch or not.