I don't understand either. You are talking about logging as PIC whilst the other pilot is just a passenger. The fact that one of your passengers is a pilot has no bearing on what you log.
There were FAA guys logging IFR pic and safety pilot time and that may be where the confusion comes in. If it's VFR and you are not logging the same hours then I don't see where the problem is.
By the way the hour building portion is there for a reason not just an obstacle to get over in the quickest and cheapest way. It is there to turn you from raw PPL to an aircraft commander. Do as much varied flying as you can and push yourself to expand your comfort zone and abilities.
Drilling holes in clear skies as quickly as possible is not what it is about.