Working overseas it was the companies Airport Manager who decided whilst the doors were open - only became the Captains aircraft once doors closed.
As PIC I decide when they board because I am now responsible for the passengers.
The position of the door does not change the responsibility of the PIC under the Act.
Out of interest , if there was a delay of 1 hour for bags to be loaded but main cabin door closed after cargo door would the delay go to baggage or late boarding of pax ?
You answered your own question, delay due to late loading of bags.