I suppose the first point is, who'd fly in it?
All nuclear-powered ships are military, hence their "passengers" don't have too much of a choice of whether to sail in a nuclear-powered vessel, other than not signing up - unless they're conscripted of course. Besides which, everyday airline passengers aren't trained to deal with emergencies on board, military crews are.
Add to that the inherent dangers of flight (what goes up must come down) as opposed to sailing the high sea's (what floats will float, power or not) and you're gonna struggle to get willing passengers.
Now there may be a "market", for want of a better phrase, in freight ops, potentially even replacing sea-freight to a large extent if it was cost-effective.
I suppose the other problem is that if there was an accident, the footprint of the accident in a current airliner would be measured in about 1-10 square miles. A nuclear airliner, well you could have an accident footprint of 100's of square miles.
On a technical level, I'm not sure how you'd convert nuclear fission into thrust in an airplane. Nuclear ships rely on steam turbines to turn a screw at maybe 50 rpm (at a guess) - so you'd need to find another method of thrust that would be powered by nuclear fission/fusion, but capable of outputting an awful lot more rpm.
It's an interesting exercise all the same, thinking through the why's and why-not's of the argument.