Maybe you could see it as an opportunity rather than a disadvantage.
When I learnt at Portmoak as a student in the early 1980's I became a member of the Tuesday group which consisted of a number of older retired members (including instructors)who got together on a Tuesday to fly rather than at the weekends. I forget how I got involved but it was probably a case of discovering that I had no lectures on a Tuesday and just asking. It seemed a bit odd at first but it quickly became apparent that we all had a common aim and everybody just mucked in. I had a great time, made some good friends and learnt a lot.
There may be a similar group you can get involved with rather than flying at the weekends when it is busier.
Curiously I drifted away from gliding to boating and discovered that my local boat club has a similar Tuesday group - the only difference is that I am one of the old ones now doing what I can to help the younger members. That said I am planning to be one of the ones described by Mary who drifts away and then comes back now that the children have left home, mortgage paid off etc.
Wherever you decide to go don't be put off by first appearances. As I said at the end of the day people are just there to go gliding and enjoy themselves.