That's incredible!
Let's ignore his breaking the 500' rule for his low pass across the runway. We'll give him the benefit of the doubt and assume that you misjudged his height, since that's a fairly standard defense when accused of low flying. (Note that I'm not suggested for one second that you did mis-judge his height - I'm just giving him the benefit of what tiny amount of doubt there may be!)
Did he offer any alternatives to your suggestion that you backtrack? Maybe you should have just stayed where you were? Or pulled off of the manoeuvring area while he landed?
Since you were on final approach before he was, you had right of way, and you maintain that right of way until you vacate the runway. Common sense clearly says you were right, and it seems that the law agrees with common sense on this occassion, too.
I'm curious - what did you do???
FFF
----------------
(I was once landing at an uncontrolled airfield in America, called final, only to hear, about 30 seconds later, an aircraft call that he was "crossing the runway". I called "very short final" for the same runway, and the other pilot - if we can call him that - replied "Yeah, don't worry, I'll be well clear. And how long have you been practicing the English accent for?"
This was followed by one of the fastest runway-crossings I've ever seen! My hand was on the throttle ready to go around - it wasn't necessary, but it was close!)