BRL
I have been up with lots of people and all appear to have really enjoyed the flying; I never had any problems, and that includes my children who have been up a number of times. Generally I prefer to fly with somebody rather than alone.
However the sole pilot is responsible for the plane and for getting the whole lot safely back and that applies regardless of any passenger emergency; adults in that situation would be left alone to look after themselves but kids cannot.
If you are a single adult in a plane with children and something happened, then you could have a problem dealing with it.
Obviously it depends on the kids' maturity, and there is only one way to establish that. But there is no doubt there is a lot more responsibility involved. A number of GA accidents (obviously only ones where somebody survived to tell the tale) have been linked to pilot distraction by passengers; I recall reading one report recently where a pilot appears to have completely lost the plot due to passengers being sick, etc.
I find the biggest problem by far with passengers is something completely different: you (the pilot) may have a few days to spare, but they need to get back to work the next day and you cannot risk a flight in anything resembling suspect weather. On two occassions I nearly bought an airline ticket for a passenger (the cost would have been huge) but in the end we flew back the next day. Such is life if flying abroad without a full IR...