Yep!
Also any video files. There is actually a "pack and go" option under the file menu which copies all the files you need to run a presentation into one place and ensures the psaths to run music, videos, etc, are correct - it also includes a copy of PowerPoint Player so the presentation can be run from a CD, etc, on a PC without PowerPoint.
Cheers
Treadders