It goes away as you fly more. Promise. Used to be the same. But challenge yourself and fly to places you haven't been before etc. Build confidence, expand your skills.
Today I experience the same thing to a lesser degree as a new IFR pilot. Doing approaches to minimums or encountering icing can be nerve wracking, but as I build up my skill and experience in this area, I can feel the nervousness diminish.