For recreational flying, I would avoid taking other people's kids younger than six or so. That said I took my own kids younger than that (my youngest at 18 months) simply to get them used to the fact that airplane is a family mode of transport. Liking it was not an element, though they all do, even into their 30's now. An exception to this was many years ago visiting family in Florida. The older brother really wanted to fly, so I took him. Mom then asked if younger brother (3 years) could go for a fly. I turned to him and asked him where his brother had just been. He pointed at the plane, then arced his hand around the airport circuit back to the plane. Yup, he understood what flying was, so I took him, though just for a quick circuit.
I once did agree to taking a mentally challenged young fellow. Unfortunately, I probably should not have with the right side control wheel in. He decided to fly, and I landed the plane fighting him on the controls, as he refused to let go. It worked out, but was messy. I just could not convey that he had to let go. Gently swatting his hands made him more eager to hold on. Now, new passengers don't get a control wheel at all, unless I want them to have it, and I install it for them.
Risk/benefit analysis, do you want to be in the sky with someone else's kid if they loose it up there? What's in it for you, if their first flight becomes their terror, and your difficult flight? At the least, assure that you have control of the length of the flight, and can come right back - no huge traffic pattern airports!