I'd agree with Richard. It's worth doing occasionally, especially if the drive is nearly maxed out, but it's nothing to worry about.
I have seen NTFS get itself into a huge mess on one XP PC, where a drive was used for a long time with only 100 - 200 Mb left. Lots of downloading, installing and deleting to clear up a few hundred megs for the next install. The Windows swap file was also dynamically sized on that drive, so it was continually looking for and releasing space. Eventually swap file performance slowed to a snail's pace, and the defragger couldn't clear up the mess so we ended up doing a format and reinstall.
That's a bit of an extreme example though
