Good advice from all contributors so far, here's my two penneth.
Remember that the max cable distance without any significant performance loss is 100m (inc patch leads each end) which is a heck of a lot when cabling a building. You would do better wiring it all back to a central cabinet and having a 48 port switch. Having one switch would also make any VLAN's easier to manage, it will also eliminate possible latency issues which IP phones can't abide.
Cat5 will be fine unless you are running VHD cameras or moving lots of very large files on a very regular basis.
If you have to run it in the layout suggested then use Cat6 links between the switches. DO NOT use Cat 5 or 6 links between buildings that have a different mains supply (different phases of the same supply are usually ok). This will cause performance problems and potentially serious safety issues (and hardware damage) should an earth fault occur on the mains supply. If you have to run in this scenario then you should use a fiber optic link with a media convertor at each end.