I don't think it should be an issue really. There seems to be a feeling that once you're a pilot then that's it, you should carry on regardless. I used to like brandy, can't stand the smell of the stuff now so I don't drink it. I don't loose sleep over it. If you don't want to do something then don't do it.
I mentioned in the gliding thread that I've just rejoined my old gliding club. I know after around a year I'll have got the gliding bug out of my system and will pack it in for another five years or so. I may even never do it again, I don't know nor do I care really. If I want to do it I will, if I don't I won't. There's no soul searching to be done.