As with anything else, you will get the excrement scared out of you once in a while. Generally speaking, fear (but not panic) helps to keep you safe.
When you should be concerned is if turbulence on final does not make you worried. Just remember to make space on your flight bag for a change of underwear and you'll be allright.