I take it when you say "manned tower and is currently under IFR conditions" you mean that the airfield is in IMC.
I think you might be more correct to say that the conditions at the airfield would not permit VFR flight. An airfield is an airfield and meteorological conditions are what will determine whether the pilot is able to fly VFR. ISTR that in the UK rules, where there is a CTZ, the wx report at the aerodrome shall determine the flight rules that a pilot may fly in accordance with within the CTZ - this may change with the implementation of SERA.