Set the PB when stopped, everytime, anywhere. 4 aircraft types later, I've yet to have a brake fail to release - and never heard of anyone else either who got stuck.
Stop the aircraft, set the brake and free yourself up to look at the bigger picture without worrying about moving inadvertantly.
The risk of a brake failing to release is so infintesimal that it's irrelevant compared to the potential threats arising from not having it set.