Paul,
I remember becoming quite panicky one day when I had just started doing solo circuits. Talking to our CFI afterwards, he thought that the combination of only slightly challenging conditions (they were safe - probably better than recent weather) and my low hours caused a certain about of disorientation and, thus, panick.
Take things slowly. This is definitely a hobby where you have bad and good days. Even at 60 hours, I still found myself trying to steer a PA28 whilst taxying with the yoke one day!
It really is like riding a bike or driving a car though - as Whirly says. You focus like hell for ages. Then you stop thinking about it and suddenly you can do it, wondering what all the fuss was about.
Keep at it.
HH