It's a hard one to judge. At my airline we allow passengers to sit at the exit rows with an hearing aid. It's logical as an hearing aid is a correctional tool in the same vein as a pair of glasses yet you would never move a passenger with who wears a pair.
We see glasses in a different light (no pun intended) which is why Hearing, sight or other disability is a no no for exit seating in my manual with a caveat for hearing aids but not for glasses.
I had a difficult call to make just the other week. A young guy walked onboard in shorts. It wasn't until my number 3 informed me that he was sat on the exits that i realised he had a false limb. I had a word and because he walked onboard just like any 'able bodied' passenger with no limp, good pace and without sticks, I was happy to let him sit there. I couldn't see any guidelines in the manual for this scenario.
All that said, maybe the airline in this case doesn't allow passengers with hearing aids to sit there. If so the crew were just doing their job. To take out your hearing aids was just a bit childish (imo) and I definitely wouldn't allow you to sit there.