Don't trust 'em!
As someone whose very livelyhood depends on transferring completed tests off student's computers onto a memory stick, I invariably unplug a memory stick before taking it out. Notwithstanding this, I still manage to occasionally "lose" files to corruption (three students' tests as recently as last week).
That could be down to a dodgy memory stick (I have now "retired" the stick I was using last week), but it could also be down to my impatience - sometimes XP is remarkably reluctant to "let go" of a memory stick - you need to ensure that it doesn't have any open files and/or folders, particularly if you have Windows Explorer open and anything on the memory stick selected.
You do, very specifically, need to unplug the stick if you should be using Windows NT (managed to lose 170 tests through not doing that), and to a lesser extent with Win 2000.
You also, very specifically, DO need to unplug external USB hard drives before removing them because Windows quite often seems to delay writes to an external hard drive. I have also discovered that with external drives that use an external power supply, when connecting them it is best to plug in the USB cable before you plug in the power cable. That seems a tad unintuitive to me, but doing it the other way round (power cable first, then USB cable) has occasionally resulted in the device being recognised as an "unknown USB device" without allocating it a drive letter. I must admit that this only seems to occur with Win 2000 and not with XP.
Bottom line is that I strongly think it is better to unplug a stick before removing it. And where possible, copy and paste to a memory stick rather than cut and paste. Convenient they are, but I don't trust 'em. And, BTW, right clicking on the drive letter and using the "eject" option very rarely works - click on the icon and unplug the device from there.