If in, doubt, check: under Windows it's a setting on the drive itself. On the "Policies" tab, it's these settings (in XP):
With some devices, that dialog will be greyed out and locked at the top option - in other words, Windows knows it's the kind of reader where you can yank the card out at any time. Still, personally, I always try to safely dismount disks: no exceptions. Even if there's no OS-level caching going on, it's still possible that an application is writing to it.