To an IFR you can order to climb, descend, turn... anything. With VFR you canīt. Thatīs a big difference.
Also, in C airspace you have to provide separation between IFR and VFR, and thatīs always difficult as you canīt really control the VFR. So YES, VFR are more a problem than IFR always in C airspace (not in E).