Having flown Airbus for many years, I do scan the memo, and as you know, the FBW and cockpit layout is a very well thought out system, and also reminds you if you apply power with speed brakes still out.
I have flown other types though, including Boeing 737 classic, which does not have very good annunciations, so keeping a hand on the lever is a good insurance policy against distractions.
I deploy the speed brake with my inboard hand, which is the same hand I use for adjusting the FCU. It’s not the end of the world if I have to take my hand off the speed brake lever, but I don’t unless I have to.