Some years ago I was exiting a large complex fire in California (USA), in a C-130. I was following established routing in and out of the fire area, and had just confirmed my transponder code with ATC. I spotted a Brasillia approaching from the right and above, and he passed close, quickly disappearing into the smoke below us and to our left. He was close enough I could clearly tell the captain was wearing Rayban Outdoorsman II sunglasses.
The Brasillia, flying for a regional airline, was talking to the same controller we were. He was operating under IFR, we were operating under VFR, albeit in low visibility conditions. The encounter took place inside a Temporary Flight Restriction, for which we were an assigned party.
No matter what the circumstances, IFR or VFR, everone is always responsible for seeing and avoiding. There is never any substitute. You can enhance your ability to look by using TCAS, TPAS, ATC, etc, but there is never a substitute for constant vigilliance outside the cockpit. Had I not seen the Brasillia and taken evasive action, there's little doubt that we would have collided. We didn't miss by much, as it was. You don't need to look away for long to miss the whole encounter; even at the low altitude speeds at which we crossed paths, the encounter came and went very quickly. It's really something you don't want to miss.
The same applies, even more, in the traffic pattern or near an airport, navaid, or airway. That's where airplanes gather. The airport in particular will find most pilots and instructors alike looking at the runway, focusing their attention, and perhaps not being as vigiliant as they should be. Don't let that person be you. Eyes up.
The question still arises. Who has more responsibility? The bloke looking forward and descending or the bloke straight and level? How anyone can descend without having a look first I don't know.
Who has more responsibility? Everyone.
Look for traffic like your life depends upon it. Because it does.
Descents and climbs should be punctuated with clearing turns to ensure there's no traffic one is overtaking, or which is overtaking one from beneath or above. In level flight one should make frequently clearing checks around the aircraft to view blind spots. Same for turns. Know who's out there, and don't let the radio spot traffic for you. It can certainly help, but nothing replaces what you can see.