In the same way you can never be considered an ex smoker I think you can never be considered an ex flyer.
Flying is hard and I've tried to give up plenty times. But after a couple of weeks away the itch comes back and it just has to be scratched.
It took me a year of weather to get the PPL and I'm very glad I stuck it out. One flight in a beautiful summers evening or a crisp cold day is usually enough to compensate for a years worth of weather, technical faults, instructor and flying school issues.
Your situation might change in the future, it might get easier and do you want to have to buy the equipment all over again?
Whatever your decision good luck and as you point out you have got further than a lot of other people.