In other words, if (for sake of argument) 1 in 20,000 people take the wrong flight, does that really justify the process of double checking?
I think the second half of this sentence disproves your point. Say 1 in 20,000 was the correct firgure. This is so low because these things are double checked.
So to answer your question, yes.
Is holding your boarding pass for that extra 10 seconds that it takes you to walk from the gate to the aircraft THAT big a deal?