Doesn't just happen in bad viz, either - although obviously that's when it's more likely to happen. If the viz is anything other than perfect, I always try to get a RIS if possible - except that everyone else does the same, and the controllers are often too busy. (Also, from your description of your flight, it wouldn't have been long enough to do that.)
The only way I can justify this is with the "big sky" theory. Two cars driving down a single-lane road in different directions, neither looking for traffic, will obviously hit each other. Move those two cars to a desert, and there's a reasonable chance that one will pass in front of the other. Now add a third dimension, and the chances of a collision are actually pretty slim. Worked for me on Saturday, returning from the PPRuNe fly-in at Duxford to White Waltham, did an overhead join, checked the published circuit for traffic before joining on cross-wind, but completely failed to notice the aircraft which was flying a very tight circuit, well inside the published circuit, until I was on top of him. Literally on top of him, by about 50 feet. That third dimension can be very useful sometimes! And I've also learnt to check the entire area for aircraft when joining, not just the published circuit!
Stay safe,
FFF
--------------