Since I've been in aviation, both in Permit and CofA types, the most frustrating thing I've found is in maintenance.
CofA types are more expensive (my prop was 20% more than the identical Permit equivalent) because of Form 1.
I've been ripped off by maintenance organisations over the years whether the aircraft was Permit or CofA - the worst was an LAA inspector, surprisingly.
I'm pleased to say my current CofA arrangements have been great. Delays do happen, but so do they with certain inspectors.
I've also seen some seriously dangerous Permit aircraft - one had to be transported to its new owner by road as it was in such a poor condition and another, after its delivery flight was grounded on the spot. I have little doubt it occurs with CofA aircraft as well.
But these days I would be very wary about buying a homebuilt or kit aircraft.