It is no more than a mild irritation though and becomes almost imperceptible when you become accustomed to it.
Personally, I never fly stick trim off, even ASE out as I much prefer a 'datum' position to work around rather than a floppy stick. It helps prevent overcontrolling, especially IMC.
I was flying a Wessex yesterday which has no ASE at the moment and with stick trim on you can at least relax your grip on the cyclic a little.