An airport operating the IFR rules properly (typically, one in Class D) will not let you depart VFR into sub-VFR conditions, so there you have to do an IFR departure, and the existence of an IFR FP will be irrelevant.
Don't forget Special VFR. Some aircraft, such as helicopters, routinely operate SVFR departure / climb to IFR outside CAS / IFR or VFR transit & letdown (or an SVFR arrival). Quite routinely, departing from airfields in an out of Class A or D airspace. No requirement for a flightplan, only an ATC clearance for CAS.