You sit in the left seat, flying the airplane, plan the flight etc, and agree w the other pilot that you have the responsibility... That makes you the PIC (P1).... If he logs safety pilot under FAA in his log book, that will be wrong, since you have not been under the hood... But what he puts in his logbook is not you problem. You are legal....
When 'safety pilot' is ALL the other chap is logging it isn't a problem. When he is an instructor and logging 'In Command' then it is. It's the 'In Command' bit that is key.
You can still hour build with an instructor logging PIC, but only as P/UT.