Intruder,
IFR and VFR might be mutually exclusive, but IFR and VMC aren't. If you are actually flying IFR then you do not need a safety pilot, you only need a safety pilot (UK rules anyway) whenever you are simulating/practicing flying IMC but are actually VFR.
In my Jepp JAR logbook there is no provision for logging of Actual IF time, just for IFR time. Whilst doing air taxi work 90% of my flights were IFR, but a lot less was spent in IMC.
I am relatively new to the game, but I think it used to be the case that IFR time didn't matter, and honestly this makes more sense, because in the UK it was (or still is for a while) legal to log IFR time without any instrument qualifications or training, as long as you were VMC. Under EASA regs they don't seem to care about how long you have spent flying with sole reference to instruments, but ask you to log time spent IFR. This is either already or will soon be illegal in the UK (I think other countries in Europe already state this) unless you hold an IR of some sort.