Many ex pat Brits are "elderly". They are a greater drain on health care services than those EU citizens who come to Britain not to retire, but to work and/or study.
The Aussies are on to that. When my niece moved there in her early 60s to live with her daughter and family who had emigrated earlier, her resident's Visa cost her something in excess of £30k. Basically an advance payment on healthcare she may need in future years.