I think you should have a few more, as it becomes less alien you will relax, the fear that seems to freeze the brain will go, the work load will reduce and you can divert some brain power to the pleasure centre
If you think about it, the 1st time you do anything, especially something that is important to you, there is a degree of nervousness.
I was taught by the RAF, my instructor said there was no such thing as a natural pilot, some people pick it up quicker than others.
Don't rule out changing instructors if you need. Human nature being what it is some instructors will be better at dealing with those that are nervous than others.
And as for turning up feeling that don't know anything, you'll probably find most of the pilots there know are darn sight less than they should!