We need more information. Broadly I support BOAC's point that the word refuse is likely to get you in trouble with an employer because it smacks of unreasonable behaviour and that is tied up in basic expectations of a contract.
To decline to extend a duty because it is not safe is very different and should be supported by the CAA and BALPA.
More Info anyone? Rgds BP