My bank manager, mortgage advisor, and every other person in the world take my "allowances" into account when looking at my pay slip for everything else.
Apologies if I have misunderstood, but if you have been including your allowances in your declaration of income on items such as mortgage applications, then I suggest the problem is of your own making, albeit indirectly. The tax man is not stupid and if people go around doing things like that, then sooner or later they're going to catch on.
Per diems are
not income and should not be treated as such. My per diems only just cover my expenses, (although I do stop over in one of the most expensive places in the world).