I have always thought that "go crew fatigue" is rather a dumbed-down phrase, more often (but not exclusively!) used by the less well informed than by professional Flight Crew. In my opinion, it certainly contributes nothing towards properly describing the real situation.
It is a fundamental legal responsibility to ensure that you are properly rested before flight and the Operator's FTL Scheme should always be a good guide. However, as a professional, the final decision is yours. On the few occasions where you have to exercise this discretion, it will surely be because of "insufficient or inadequate rest".
Neither are you "letting the side down" - such decisions are simply one aspect of the overall decision-making processes required by the privileges and responsibilities of a professional licence holder.
JD
