In the USA, FAR 1.1 defines PIC as the person who “has final authority and responsibility for the operation and safety of the flight; has been designated as pilot in command before or during the flight; and holds the appropriate category, class, and type rating—if appropriate—for the conduct of the flight.” I bolded "or during" for obvious reasons.
If I can be bothered, I'll dig up some Australian provisions and precedents for PICs changing during a flight.
If I think there's a substantial risk that the person in the right hand seat is going to bend my aircraft or me, I'll be taking over anyway. Fortunately the probabilities of that happening are extraordinarily remote.