Agreed. Even if we can spot another likely aircraft, how can we be sure it is the actual target of the RA?
Answer: We can't.
TCAS looks at all transponder equipped aircraft up to 80nm away
including behind and to the sides of us and up to 9,900' above and below our own aircraft.
I might be able to spot aircraft in front of me, or a contrail, but I cannot see aircraft behind me, or to the side - behind the cockpit window field of view cut-off, or those directly above or below me. So it is very dangerous to assume that the one target one can see is the source of the RA.
An RA, incorrctly ignored on the basis of observation of an aircraft ahead might cause severe problems if the actual intruder was
behind, directly above/below, or otherwise out of sight of one's own aircraft.
Don't even bother looking. Instead, if a TA is triggered; eyes inside onto the PFD and mentally review the actions and calls for an RA.
