Yes, you are quite within your rights to refuse to drive if fatigued as per the Health and Safety laws. You may also collect your P45 the next day!
This serious matter was discussed on pprune a while ago with regard to flying and fatigue. If you subscribe to this concept, then the fear of losing your job apparently will motivate you to risk your life and that of others. On the road we are probably talking significantly lower numbers at risk, but still the risk is there.
You have to have solidarity on this matter though. It bites at the heart of air safety and road safety. You can work on it in ways other than just living in fear of losing your job, and therefore doing
anything your management requests. It begs the question: what
wouldn't you do for them?