It depends very much on how interested the family is in flying.
I've taken all mine - even mother and mother-in-law, now sadly long-departed. Most of them were either bored stiff or petrified. Mother in law thought it fabulous - we flew over the local area and she was thrilled to see her house and all her friends houses from above. I was amazed at how well she recognised places she'd never seen from that angle.
Daughters won't fly with me - it's too boring. Wife will, gladly, if we're going somewhere interesting. Even then, there's a threshold of pain at about 2½ hours airborne time. She won't wear a headset cos it's claustrophobic.
So - if you feel confident that you can handle it, take them - one at a time - and see how it works out. If they fall asleep, they aren't afraid.