Don't forget too that typically the IR is tested on a twin engine aircraft whereas the IMC is usually tested on a single. Not that flying IFR in a twin is difficult as such, but its a bit more to cope with, and in the flight test one of the approaches has to be asymmetric which can be demanding, especially in turbulence and with marginal performance on 1 engine!
Perhaps another difference is that due to the need to enter an Airway, you have to climb higher, and the descent will take slightly more planning than from lower level in a single engine.
No single part of the IR is difficult, but putting the entire thing together in a competent manner and maintaining your scan for 2 hours whilst dealing with nav, RT, engine failure, weather, icing, fuel, timing etc requires some practice.