Perhaps SFO tower could have mentioned the GA traffic when issuing the takeoff clearance.
"The GA guy has you in sight and is staying clear" means that he has your and your passengers' fate in his hands if you do not yourself have him in sight.
We do like to feel in control of what is about to happen and in this case the FO did not.