Nothing wrong with giving up now. Only if you have any desire to do so get an FI rating and instruct when you feel like it
. You should definitely complete an IMC course though because of the benefit. That is all of course if you plan on continuing to fly but honestly if you have a job and you're happy enough it wont make you any happier because you have 20k to spend just to get you to the point of being employable and then you will almost certainly have another 20-30k to spend on a Type Rating if you want to have any chance of getting a job in the next 5 years.
£50k is a lot to gamble on maybe enjoying something?
The world is full of opinions and asses.