I think it would be rather difficult to persuade the CAA and the court that my 500hrs and private ticket allowed the ATPL with 10k hrs sitting next to me to walk free because he was a "passenger"
Then you don't understand what being designated as Commander of an aircraft actually means. The CAA are very clear what it means, and that is that there is only one Commander of the aircraft at any given time.
It is not based on any seniority or experience gradient, it is based on who accepted responsibility before the flight to be the Commander. It is of course possible to share a flight and be Commander for a portion of it, but you would need to agree this between the pilots involved beforehand, and also complete any pre flight documentation to show this to be the case. In the abscence of that, whoever signed for the aircraft is the legal Commander. (This is in a flying club environment with a PPL flying the aircraft). The legal Commander has responsibilities placed upon him or her for the safe conduct of the flight, which can include refusing to fly if unhappy with anything.
Of course, whether this is actually explained to everyone who gets their licence is another thing, but alas ignorance of the law is never seen as a defence.