Might well be a connection with the State Pension if you are receiving one. I guess you are as you mention not paying NI.
HMRC cannot deduct tax from State pensions, which are always paid gross - with no tax deducted - so you get the full Monty - about £5.8k. But if you over the threshold to pay tax, you still owe tax, so it will be deducted from another pension or pay.
£50 a month is about 10% of your pension, and 10% is the initial rate of tax .......
Personal allowance is £10.6k, after that the first £5k is taxed @ 10%, so if your other pension is about £9 or 10k, then you'll be taxed @ 10% on the rest - which is about your £50 a month! Simples.