So long as you are "sole manipulator of the controls" and your friend is not logging the time, then you can log the PIC. As IO says it may get sticky from a group / insurance POV.
If this is a common practice, you could ask your friend to approach the group to add you to the insurance and offer to pay for any additional premium and then your friend could guarantee you - If this came up in our group I'd agree to it because I'd trust the rest of the owners to vouch for someone else, and as long as they are only flying with one of the owners then I wouldn't see a problem.