If you click click the autothrottle on every flight, you will not make a mistake so quickly by inadvertently pressing toga instead.
It all boils down to:
1. Are you allowed by sops?
2 are you comfortable to do so because of your training?
3. Are you encouraged to manually fly the beast by the airline/crew mates/culture?