Why giving up IS a big deal! Quitting would be a big deal.
Because if you are a pilot, you have made the grade. (funny how often I manage to mention this fact to people I meet....yes, I am a pilot. I am exceedingly proud to be a pilot. I will always be a pilot, even though I can no longer fly solo or instruct)
So don't quit, take a break. Family comes first. Dont insist on taking the sprogs flying, they get sick and bored.
You can always come back to flying, even after twenty or thirty years the skill is still there, rusty but renewable. When the kids are leaving home, (if they ever do!) more time, more money, more opportunity. If they don't leave home they should at least pay rent, so they'll appreciate the roof over their heads!