My advice would be to give it a go. There are many of us who started out doing a PPL on the weekends while holding down a stressful full time job. The only downsides to this course is the British weather which can interfere with numerous weekends and the extended time it may take to get finished. On the plus side doing it this way allows you to get a feel for your aptitude and to make a solid decision about whether it is a career choice you truly want to make.
This is the way I did it back in the late 80's and after a year or two of private flying decided to make the career change. There was then and there is now a huge risk involved as nothing is guaranteed but it can work out.
Twenty odd years later I am flying for a large International corporation and leave in 2 or 3 weeks time for type school on the Gulfstream G650.
If it is a dream of yours then be brave and go for it. It is no good looking back in 30 years time regretting what might have been.