There's nothing in the ANO to support 'VFR flights only'. You only need specified equipment to fly IFR in CAS
Sorry, but that is not correct. Look at schedule 5 to the ANO and at the table under paragraph 2 of the Schedule. An aircraft requires "Radia navigation equipment capable of enabling the aircraft to make an approach to landing using the Instrument Landing System". This applies to making an approach to an aerodrome "... notified for the purpose of this sub-paragraph".
AN84 specifies non FM Immune instruments be limited to VFR only. That means that if you are making an approach for real in non VFR (i.e. IFR) then you must have FM immune. To stay within AN84 you would have to use the equipment in VFR, which defeates the purpose of the ILS other than in cases of training. The question of controlled or uncontrolled airspace is irrelevant to the use of a VOR/ILS, though would be relevant to the use of the VOR in uncontrolled airspace under VFR, which is permitted.