As far as I can tell, you were asked to sit by the window and centre seat and the aisle seat was occupied by someone not flying with you. If that's the case, the reason should be that because you were sitting with an infant on your lap, should the worst happen you wouldn't be blocking the exit of the "stranger". Infants are not allowed to sit on an aisle seat when there are people in the row not flying together. If everyone in the row is in the same family/group, then the infant can sit on any adult's lap.
Normally four masks are fitted for each three seat block. This extra mask is for infants of people in the aisle. Some a/c have three on one side and four on the other, and then cabin crew have to move and infants sat on the 3 mask side.
Hope that helps