Is the onset of the fear closely associated with certain visual images? For example, is it consistently associated with cloud layers?
It's possible that certain images remind you (unconsciously) of a bad experience in the distant past. Each time the visual imagery around you in the aircraft approaches the images you remember from this bad experience, you feel anxiety. It's not really a full-blown phobia, but just a bad association. The more specific the situations that upset you, the more likely I'd expect this scenario to be. For example, if being between cloud decks panics you but nothing else does, it sure sounds like it's reminding you of something.
I'm reminded of a case in which a woman had a very strong and irrational fear of bold vertical lines. She couldn't tolerate seeing such a pattern anywhere without feeling a bit of panic. One day, by chance, her uncle mentioned an incident that occurred when she was a toddler, in which she tried to put her head between the bars of a railing and got temporarily stuck, leading her to panic. When she heard this, all the memories of that traumatic experience suddenly came back, and after crying a bit over it, she discovered that her phobia about vertical lines (which presumably reminded her of the railing) had disappeared.
If this is the problem for you, you need to find some way of isolating the incident from the past that makes you nervous when you see certain images while flying. A psychologist might be able to help. The anxiety could also come from something other than the visuals, but given that visuals are such a large part of flying, I'd vote for the images first.
In any case, if you enjoy flying and wish to pursue it, seeing a psychologist to try to find the cause of your occasional panic attacks would seem to be a good investment.