It is their decision, as the commander needs to consult every crew member (at least under EASA). Not to mention that it should be a proper 'airmanship'.
I respectfully beg to differ. It is called 'Commander's Discretion' for a very good reason.
To protect individual crew members most OMA's have a phrase such as this inserted. 'No crew member shall fly if they feel that they are not fit to fly or are likely to become unfit to fly during the rostered duty period - including any possible use of discretion.'
If you feel unfit on the ground during a (delayed?) turnaround flight then inform the captain (who will arrange appropriate medical assistance) and be properly stood down with all supporting documentation available. This is nothing whatsoever to do with discretion. It is to do with fitness to fly. If you feel fatigued/unfit halfway around a duty then you should inform the captain who will arrange appropriate medical assistance and may then be able to manage the situation depending on crew numbers.
But it is 'Commander's Discretion.' That said, it is a very brave (or cavalier) Commander who ignores a crew member saying they are unfit to fly. In flight, stand crew member down immediately, PA for a doctor etc. On the ground, immediate medical assistance. Whether to utilise full or part crew discretion or not is the sole decision of the Commander after he has performed HIS assessment of the crew's fitness for discretionary duty. I have seen it applied both ways. On one occasion the entire crew were totally fatigued but the cabin crew wanted to get back to main base (for personal reasons) and it was the Commander (who also wanted to get back for the same personal reasons) who quite rightly said 'No, it's not safe.' Gave his reasons in a report and the company supported him.
One other tragic occasion occurred when cabin crew operated when probably they shouldn't have on a night turn from UK to a holiday destination in the Azores. Flight operated without incident but cabin crew member fell asleep at the wheel while driving home and was killed. If you feel you are not fit to fly, then don't. Please, don't! But back it up with a medical report and don't confuse it with discretion. Any reputable employer will support you. Sadly, there are less and less reputable employers around nowadays.