Papi - are your kids under 12? If so then I'd see that as a serious problem. Otherwise, I suppose it's an inconvenience and it is the risk you take flying loco.
I'm really surprised Cabin Crew didnt assist and shuffle people round. Even if time on the ground is tight, it could be done in the cruise.