Is it the case that an aircraft operating under a Permit to Fly cannot be flown as PIC by anyone other than the owner(s) ?
No. The permit does not specify who can fly the aircraft in normal circumstances. However, a permit to test will often specify an individual pilot or pilots who can fly the aircraft on a test flight.