I say it's the flight plan/itinerary because I cannot think of any other rule or condition that applies to all IFR flights and to no VFR flights. Can you?
IFR position reports? CVFR fligts or VFR flights on IFR air routes make pretty much the same calls. An IFR flight within radar contact no longer has to make those calls unless asked, and a VFR flight still has to make the calls if asked.
IFR obstacle clearances? I could scrupulously adhere to IFR obstacle clearance rules while VFR, and some company ops manuals require similar clearances. An IFR flight that ends in CFIT was still an IFR flight.