As someone who has 7 years experiance in both camps, I can see both sides. I worked in baggage untill I had 2000 hrs. Flying is more mentally demanding, throwing bags, amongst other things is more physically demanding (great for keeping the weight off - I;m 20kg up on my bag throwing weight.....I should just say fatter).
If I could, I'd still do a couple of shifts each week, its cheaper than a gym membership and actually its quite a challange. Especially if ur leading a crew of 10 on a regional ramp.
By the way, on one of my teams a few years ago, between us, we had 11,000 hrs, 7 ME-CIR's, 4 ATPLS. An old groundie who never flew or even finished school, but had 40 years in the industry taught me more than I could ever hope for. He had no "education", but man he could stick the tail of 400 tons of 747 within a cm of where he wanted it on a puchback.
Different work, different skills, but both have the same strong desire to get the job done. Both depend on each other.