Domicile has nothing to do with which passport you hold.
From HM Revenue:
4.1 Domicile is a general law concept. It is not possible to list all the factors that affect your domicile.
4.2 Broadly speaking, you are domiciled in the country where you have your permanent home. Domicile is distinct from nationality or residence. You can only have one domicile at any given time.