If you want to fly more than anything else in the world then I'm sure you will make it happen. Many successful pilots had similarly arduous beginnings. Others have had it easy all the way though. I think it is all down to luck*. The main thing is don't give up and don't regret. If you stop pursuing a flying career it will not happen, but other great thing could - you've obviously got determination, confidence and intelligence to have got this far and they are quite transferable skills! Flying can be quite a fun hobby too if you succeed elsewhere. Like a Faustian-pact the piloting prize may not be worthy of the cost (in time, money, relationships, missed opportunities) and only you can decide. Wishing you luck finding your path.
*aviation is very cyclic and in my time there have been recessions (nobody could afford to fly, airlines went bust and flooded the job markets with pilots), 911 (nobody was safe and people stopped flying), "great" financial crash (same outcome as with recession), wars, COVID etc. and each time aviation comes back bigger and more hungry for pilots than before.