Correct. C of G is a very important factor, and spin recovery characteristics will drive C of G limits. But the moment of inertia can have an effect too. If you put a small mass wayyyy back, yet remain within C of G limits overall, the spin recovery could be affected, as there was more inertia to overcome for recovery. I think of a few engine change STC's which include a few pounds of lead in the tail to balance a heavier engine. That said, I expect that for spin approved GA types, you couldn't get enough mass far enough back, yet within C of G limits, to become a problem, but I would be considering it during flight testing. Happily (for maintaining spin recovery characteristics, most of the things I've tested hanging off the back of a GA plane were also big enough to have aerodynamic stabilizing characteristics to overcome any mass destabilizing characteristics.
I have spun these, they all remained compliant:
The booms are light
And:
Ugly at aft C of G, but compliant, there's quite a stabilizing effect from the stowed towed survey bird. Video here: