Please note that social security premiums and income tax are two different things (even though they usually both appear on your pay slips and tax bill).
Where you pay your social security premiums is governed by EU legislation, where you are liable for income tax is governed by bilateral double taxation treaties between countries.