I've always been under the (possibly misguided) impression that the published flight manual g-limits are around two-thirds of the calculated g at which structural failure would occur.
In my experience of military flying, flight manual g-limits are regularly flown up to as many times as are required for the training or task being done. You don't go out and go crazy with the g for no good reason (well, mostly...) but nor do you hold back when required.
This then leads to possible mismatches between the designers' assumptions and what is actually done, therefore sometimes reducing the service life of the airframe. Reduced g-limits are sometimes introduced to try and extend service life when problems (eg fatigue cracking etc) are detected.
One case in point is the Pilatus PC-9 in service with the RAAF - as an advanced trainer it naturally copped a flogging, and needed to to get the job done. However, it was soon realised that this would quickly mean the limited fatigue data available from the manufacturer wasn't sufficient to predict what might happen to the airframes long-term (bearing in mind that we Aussies keep types in service for a long time, historically).
A solution was to take a representative airframe off line, stick it on a fatigue rig which basically pushed and pulled at it repetitively for months/years to rapidly simulate the effects of the above-mentioned flogging, thus generating useful data that could then be used to work out a more accurate estimate of airframe life under real-life working conditions.
So - designers design them to a set of assumptions, then they are used within flight manual limits (barring occasional stuff-ups) and fatigue occurs at a rate governed by same.