Donīt worry about the other guys and what they think. Thereīs nothing wrong with being posh. Not everyone in the aviation are like the guys you describe. Besides, there are people like that in other industries too, especially in industries where egos are involved. I know this because I have been there.
There is nothing wrong with packing it all in because you donīt think the risk is worth the reward. You have to do what is best for you and that is admirable. Good luck to you. If you ever decide to keep flying, you will find that the environment is quite different at a flying club. The problem with flying schools and students is that the students are always comparing themselves with others. Students talk amongst themselves to try to see where they stand in ability compared to the other students. It is a normal occurrence amongst flight students, so donīt fret!
