You are certainly not alone.
I can remember the early years of having a ppl. In those days it was 5 hours every 13 months or a GFT (handling check).
I struggled to be anything other than safe.
In later years I have checked out many ppl`s wanting to hire an aircraft.
Often they would not have passed a handling check but I had to look for a safe standard (would I sit in the back of an aircraft being flown by them?)
The fact that a pilot "knows what he dosen`t know" and knows that he is not in current practice is more important and therefore adjusts his operating minima accordingly (x-winds etc).
Flying shorter more regular flights (as has been suggested) would help.
Also what you do during the flight.
It is a shame that fewer airfields allow circuit practice these days. 1/2 hour in the circuit ( tight as you can with glide approaches) every month is better than 1 hour straight and level every 2 months.
If circuits are out, then some general handling (accurate 30deg banked turns through 360 deg, then 45deg bank if you are confident). Also slow flight (stall +10kts) and simulated go-arounds at a safe height.
When you do the flight with an instructor, discuss with him/her what you feel least confident at, you will then get more out of the flight. Try not to treat it as a test
Sharing flights with other ppl`s can help. You can learn a lot from others (minor) errors and whilst he/she is flying you will be mentally flying the aircraft as well.
You can visit new airfields that way.
By knowing that you are out of your comfort zone and wanting to do something about it suggests that I would sit in the back of your aeroplane.
J.J.