There is nothing wrong with taking toddlers flying, mine have been coming with me quite regularly since they were three. Just make sure their headsets are a good fit and I can't see what damage it can cause.
My eight year old can fly straight and level and does without even trying. I took my htree year old up on Saturday and he loved it.
Be aware though that if they can't see out, they will get bored very quickly. If the a/c has clear perspex bottoms to its doors like mine, the whole experience for them is a lot more enjoyable for them.