Yeah in which case you could get an aisle seat or a middle seat?
Anyway point is relating to the case of the 'but I have a child' brigade. If the cabin needs to be rearranged then it can be by the crew in order to get kids sat by parents. A lot of parents know this and therefore will not pay as they know the crew will move the cabin for them.
If someone has not paid for a seat then technically they are not entitled to it and therefore can be asked to move regardless of whether they have a boarding card for it.
When I was crew it happened bothed on a free seating airline and assigned seating airline. Bottom line is you don't move those who have paid and you don't move your regular flyers. Simples.