You can uninstall IE, but it is a real pain as the Windows system is, to some extent, built around IE. To be honest, (and I'm a keen Firefox evangelist) it's worth keeping IE on your system as some sites - especially Windows Update - function better with it.
I'm not entirely convinced that it is ever safe to run Outlook Express - with or without Firefox. Many viruses and such are targetted at OE, not because it's intrinsically unsafe, but because it is incredibly popular. Thunderbird (by the same people as Firefox) is a good alternative.
Firefox can be horribly slow to load. This can be due to the number of extensions or half a dozen other things. If you search for
firefox speed up on Google you will find thousands of sites looking at the problem. I've been impressed with
fire tune a utility for tweaking firefox, though you can get better performance if you do the tweaking manually (but that's a real hassle).