Sidestick,
Be assured that you are by far not the only one to have done that!
Until I met airbus, I had always been on types requiring two actions to set the park brake, (eg) depress toe brakes, pull nice big park brake lever. Nobody got confused.
Since I've been on the 'bus I have commenced pushback with the brakes set; thought the brakes were "on" when they were actually "off" ( & vv); and set brakes to park while still moving slightly. Pretty careless, you might say, and perhaps there was an element of that. But in defence, I have never done any of these in 25 yrs of flying other more "tactile" types.
So, IMHO the airbus park brake switch is perhaps not "an accident waiting to happen" but it could have been designed better.