I agree. But they do't need higher skill levels.
I have friends who fly infrequently who simply wouldn't consider flying in weather/wind conditions that don't trouble me. And when they do fly, they plan it very carefully, even though it may only be a simple flight in good weather from one airfield to another 50 miles away.
They sound good people.
Unfortunately I do know others who fly on the edge of safety. In order to avoid supervision by instructors fly once a month and venture out in cr*p weather on the limit of their ability.
Similarly, I know of experienced pilots who do the same.
My own view is that if you just bang in a couple of circuits once a month and you get caught out by unexpected circumstances, you have less in your armoury to help you get safely back.