I fly a fast glass three axis microlight and love it. My hourly costs are tiny. However, every girl (and boy) that I've given a ride to has had a hard time with their nerves, despite some very sedate flying and calm airline captain style commentary from me.
Girls don't like planes, girls like the evidence of money.
As for other reasons to quit...
If you're VFR only then get yourself to a small uncontrolled airfield with no radio service.
When you can turn up as you please, pull your Cub or whatever out of the hangar by yourself and pop off for and hour to watch the sun making the clouds into ruby and custard coloured candy-floss, when you can turn the donkey down and catch the last thermal with a local buzzard, and he joins you because he saw you circling first, and when you land that perfect greaser with only the oldest club member to see you and he gives you a thumbs up, its then that you realise that the ten-year-old in you standing in the garden with the Keilkraft model was right all along.