The question who had right of way it irrelevant in this case. By definition, those rules only applies to two aircraft that are aware of each other. The Rules of the Air then prescribe what evasive manoeuvre each aircraft has to make to deconflict. (Like it would be stupid if two aircraft on a collision course would both pull-up, and then still collide despite both pilots having taken evasive action.)
The principle of VFR is "look & avoid", not "look & be righteous". If I was the only aircraft that sees the other (and the other aircraft seems blissfully unaware), my only duty is to avoid. If both aircraft were unaware of each other, it is irrelevant who would have had right of way.