As I understand it, it is because you loose any form of cross checking - You are reliant upon the remaining IRS to behave. Now if you are flying the aircraft and the IRS shows you suddenly in a sprial dive when you know you aren't you will probably do the right thing. The autopilot won't...