As a new PPL I can totally sympathise. I frequently have huge moments of self doubt. But then I have to remember that my instructor wouldn't have sent me for my test if I wasn't capable, that they have taught me everything I need to fly safely and I demonstrated that to an examiner.
I'm a little torn, I think you piled a lot of pressure on yourself, but at the end of the day, flying is flying, if the weather is good then you have less concerns. Plan the flight properly, check the airfield procedures, write them down if necessary, pre flight and get knowledgeable and comfortable with the aircraft. And ask people if you are unsure.
I visited Sherburn In Elmet for the first time last week. (Exotic I know). I printed the circuit instructions and even drew my expected join. I planned where I was going to descend, wrote my heading for OHJ from my intended position and it all worked out perfectly. I think you should go unless there's an obvious physical or weather reason not to. Ultimately if you take off into clear skies the airfield is there behind you if you feel really uncomfortable.