My father flew me as a baby in a Cherokee. I don't remember it at all, but I've been obsessed with flying since the age of 5. He did the same with my sister and she screamed so loudly he elected not to take off.
I now have a baby son, and due to aviation induced mild hearing damage to my ears, I won't be flying him until he's six or seven. The last thing I would want for him is a lifetime of tinitus, so I'll metering his exposure carefully.
I'd also like to say I think Mary is dead right, and the way to take that Baron full of kids&wife on holiday is with a safety pilot doing most of it for you. It may not be the independence you dreamed of, but at least you'll live long enough remember the whole of the trip.