Dutch, it's difficult to get a balance sometimes as you have pointed out.
However if you feel the company is pressuring you unduly... here is an idea to keep in mind.
I was in a similar situation once, wanting to offload a pax, this was a rare occasion when management were really pressuring us not to delay. All I needed to do was to speak with the Captain and tell him I was feeling unwell, and wasn't sure if I would be up to dealing with the passenger if he remained onboard. Company changed their tune pretty quick when they realised we were all about to offload OURSELVES if they made us go with this (very abusive) pax..
