I've noticed considerable differences in the speed from various YouTube hosting sites. As has been suggested either hit pause and allow the buffer to build up, or download and play later. I use Firefox with the "Download Helper" add-on which does the job.