Listen mate ..
I am 53 started flying at 24 have had several breaks sometimes 3 or 4 years , have lost friends in accidents, had kids and brought them up, in my early days I could really not afford to rent from schools so just doing minimum hours to keep the PPL. During my lay offs ( with young children mainly ) I just accepted that maybe that was that ...but the urge always returned. I put aside my guilt about spending large sums on my self as I realized it was sanity for me life changes drastically when children arrive, I took out a hefty insurance policy and joined a group with a 172 had a bit more money and flew the arse off it.
Red tape and rising costs and falling income due to starting my own business and the recession left me wondering if it was sensible to fly an hour a month ...which was what I was down to..Meanwhile my now 18 year old son spent his accumulated wealth and some accumulated from me ! on an NPPl. Knowing he was going to hate renting and not going be able to afford to do so I decided to by an LAA machine a RANS S6 with a 912 Group A for 11K..
Why in gods name did I never do this before ? I fly when I want I spend hours doing safety stuff EFATO ,PFLs at little cost £22-25 ph, a 4hr + range at 100mph, it takes two hefty blokes full fuel and 50lb of baggage, I do my own repairs and so on ....This means I can really go anywhere 2 hours there and 2 back in a 172 even group owned costs way too much ...I spend many enjoyable hours passing my tips to my son and keeping him under my wing , and now he is giving me tips! . and telling me off. Were flying to the western Isles of Scotland this year for a week and I know we will have a ball .
I have felt all the things the OP has mentioned, the key is to get away from schools and rentals ..there is far more to flying. I made a simple easy cross country last week to land for free at Boston on well kept grass a mere bimble for me ..do you know what it was one of the best ever flights, at home in the air in my own flying machine doing it right for once and even proud of my landings..because they were rather good and ever so grateful to have the health just enough wealth and the knowledge that its a privilege. In my 25 years or so of flying I can truthful say that the last 18 months have been the best, the responsibility of owning and maintaining an aeroplane was one thing, ensuring I did not pass bad habits to my son made me hit the books and brush up my flying ...how crap I was .
Yep over the years I have been skint -bored- scared to death -over confident -under confident-disinterested and let my licence lapse ....but me and flying were old friends now we can pick up where we left off...any time.
So can you OP... enjoy your family enjoy your flying it will be there if you want it if you don't no sweat .. But join a group or something.