Thinking back a bit, what scares me about potential turbulence is the behaviour of some other passengers when the seatbelt sign gets turned off. On my last flight I was sat next to someone who just unfastened his seatbelt when the sign went off. He didn't get up, shift position or anything, he just took it off. I'm not too concerned if he head-butts the ceiling or the overhead lockers, but I'd much rather he didn't land on me on the way down (up?) again. I keep mine fastened, even though I loosen it once we've gained height.