Krallu:
It depends upon what the root cause of your struggles to maintain control of the situation might be. As stated above, a simple lack of practice can leave one a little short of performing to standard, whether it be objective standards or those you set for yourself. If that's the case, it can be fixed with practice and drill. You are in the best position to know whether a lack of overall cognitive ability is the root cause. In this case, the extra practice will not yield the expected increase in proficiency and reducing or curtailing the level of responsibility you trust yourself with may become necessary before the matter is either decided by someone else or by events.
General Jimmy Doolittle made this decision himself when he felt his other responsibilities no longer allowed him to maintain the required level of skill and situational awareness. Eventually we all find ourselves lacking and face the decision of whether to step up our game or chuck it in.
If you've simply run into a low spot in your flying confidence due to an isolated case of below par performance, then you'll apply some effort and overcome this temporary setback. And perhaps learn some important lessons to apply in the future. Questioning yourself is not wrong. Best of success in getting the most from the next step.
All the best.