You are definitely being too hard on yourself.
I was shaking like a leaf on my first real lesson, even though I was fulfilling a childhood ambition.
Later, when doing circuit training, I almost gave it all away because I was having so much trouble with landings. I remember going out to the airport one day thinking that if I did not make progress during that lesson then I would stop throwing good money after bad. Fortunately my landings improved and a few weeks later I did my first solo - a most satisfying achievement.
I know many good pilots who say they struggled in the eariy stages of flying. We thought we would be naturally gifted pilots but found that there is some skill involved and it takes time to learn those skills. However you don't have to be superhuman to be a pilot (contrary to what some young pilots might like you to believe). It essentially comes down to practice, practice and more practice, and that's the case right up to the day when your flying career ends. Any airline pilot will tell you the same thing.
Have fun!