There is another way of looking at this.
Flying in good weather is idyllic. Flying in poor vis, high winds, rain etc is actually bloody miserable. For most of us it really isn't enjoyable.
We pay a small fortune to fly. Most of us do it for fun. So why, just why, go flying in weather where you won't enjoy it? What is the point?
Yes, there may be people who'll go. They may think you're a wimp. But who the hell cares? What do you have to prove?
Of course, there will be those who say you should be stretching yourself. Up to a point, yes. If you're terrified of a single cloud or one degree of crooswind, that's a little extreme. But if you don't ever want to learn to fly in strong winds and rain, then why do it? It's not compulsory. It's no crime to be a fair weather pilot.
To repeat, we do this for FUN. So why do it if it isn't fun?