And you can get the aircraft into an unrecoverable position in far less time than it takes for the other Pilot to get back from the toilet and the FA to open the door.
In fact it can be done with the other Pilot sitting at the controls. There is no improvement in safety at all, possibly only a detriment.
Why get annoyed about it? Because it is just another mindless knee jerk B*ll**** inconvenience dreamt up in the name of security by those who have no idea!