You and your family need to go abroad to avoid UK taxes.
It helps to have let your house out, and have the foreign based employer tell HMRC that you are now employed abroad.
HMRC have seen all the lesser variations, and are clamping down if they think that you and yours are still, really here.
You might need to get a divorce if your family stay here, but I'm not sure that even that will work.....