As others have said, it is the recovery from the dive that could overstress the aircraft. If you can control the speed to less than Va, then you should not exceed the aircraft's g limitations with a straight pull on the stick, although you might stall again. Over Va and you could break something.
You still need to be careful even at less than Va, because if you roll and pull at the same time you can cause damage. If you recover in an inverted attitude, roll to the nearest horizon immediately after stopping the spin, then pull.