My 2p worth....
1/ Stop carrying passengers right now
2/ Own up to your instructor. Admitting your 'little problem' to him/her will be less embarrassing than when the CAA come knocking at your door asking why you've infringed some busy Class A airspace or whatever. As others have suggested, if you've got a problem with, or do not want to admit to your current instructor, try another one.... there are plenty out there !
3/ As others have suggested, once you've had a refresher with an instructor, you should try to fly more often. If flying more often isn't an option for whatever reason, then you should consider scheduling regular refreshers (monthly, quarterly, whatever ... the frequency is a matter for you and your instructor to debate) so that your confidence levels are kept high and your skills up to date rather than go rusty.
Happy flying !