Mo, I'm pretty sensitive about this myself. Never had it happen to me yet so I might be getting it right.
Here's my tip.
Quite often you'll get pushback instructions from ATC which can distract both you and the ramp man into forgetting the brakes are still set. I make a positive practice of passing any such instructions first and finishing with the words..." and the parking brake is SET".
Don't use the word 'release' in any phrase (i.e. 'call for brakes release') unless the brakes ARE RELEASED.
Otherwise it can lead to misunderstandings.
Always use Standard Phraseology.
Speak Loud and Clear and SLOWLY...especially the words SET or RELEASED.
Don't rush. Be pedantic. If in doubt DOUBLE CHECK.
The same techniques apply to when re-setting the brakes after pushback is complete.
Oh...and don't forget to check the brake pressure gauge as well as just operating the park brake lever!
PS If you ever find yourself rolling backward, be VERY CAREFUL when applying the brakes to stop yourself.