My thoughts are that if you are flying VFR, you fly in relation to the ground, on clear days, under clouds, and if you fly IFR, you could put curtains on your windows and fly from one place to another by only looking at your instruments.
Any time while IFR in VMC you should maintain a traffic lookout exactly as you would on a VFR plan.
You might see a VFR target climbing or descending through your level in Class G or E airspace, or it might be a big yellow crop-duster opposite direction same level (8,000 ft) as was my experience about 10 yrs ago.
I never assume any other aircraft's transponder is (a) working or (b) switched on.