I'm barely into my PPL and can confirm that this almost happened to me already. We were taking off in a 152 and my instructor called for flaps to be raised in stages - instinctively my hand went to the flap lever, my eyes were on the horizon heads up and I "felt" my way to raising flap by 1 stage.
Cue my instructor shouting loudly and quickly snatching it out of my hand to lower to the correct position, luckily it hadn't had any adverse effects by the time he'd spotted it and intervened. I hadn't realised what I'd done wrong until we cleared busy airspace and he explained it to me. In fact we went to a safe altitude and he showed me exactly what happens when you go from full to 0 flap and just how much altitude you can lose.
I was also told it happens quite a lot, especially with students when calling a go around and I should always be on the lookout. Didn't realise even 320 pilots with that many hours could be susceptible too!