I think it's healthy to be concerned about collisions. Just as it's healthy to be concerned about carb icing etc - you'll be looking for the symptoms / other aircraft. My greatest concern is a high wing a/c climbing into me (low wing), or my low wing descending into a high wing. So you just have to be proactive with your lookout - really rock the wings to see if there's anything below (or above for a high wing).
When flying with passengers I always brief them on keeping a good lookout - it gives them something to do and it gives the a/c some more eyes. Most of the time in the circuit they spot traffic before I do.