Having worked in both jobs i don't think it's a case of cabin crew looking down on baggage handlers. I think because cabin crew and loaders have not very much contact as far as their jobs are concerned i think it can be mistaken for being looked down on from the loaders point of view.
Take for instance the caterers. The crew are in contact with familiar guys quite often on board the aircraft and so they are on name terms with each other.
I know from experience that it's easy to mistake the lack of communication from the cabin crew as being looked down upon but this is simply not the case!
Another way of looking at it is, from the girls point of view, the most obvious memories of their experience with loaders is them trying to get a glimpse up their skirt when operating the door. So in some cases you can't blame the cabin staff being wary as i have seen this myself on many occasions!
So don't take this to heart, and even if they do seem to be careful around the loaders just remember what they may have had to put up with in the past.
Just out of interest which airport are you based at?
Best regards
Dave