Just remember that difficult doesn't mean impossible! Depends how much you want it. Could you happily spend the rest of your life wondering if it would have worked out? If the answer is no ......... then maybe, just maybe it might work for you. But for Gawds sake, keep the day job for the time being. You might not (probobly won't) get a job on a 'shiny', but that is only a small part of what a professional pilot can do.
There are a few threads on this, try a search.
Best of Luck!