Have you tried using the scroll function on your mouse or trackpad? Slide up or down using two fingers for a trackpad, or one finger when using a magic mouse. Does that produce the same result?
I suspect that somehow you're losing contact with the scrollbar, either the mouse button does not remain depressed or the mouse itself has a troublesome connection. It might help if you could let us know what hardware you're using. If possible try this with a different mouse and see if that improves the situation.