the ground staff are responsible for making sure that anyone they allocate to an exit row meets the required criteria, however passengers have a tendency to try and move around etc when the F/A's backs are turned, but the final responsibility before take off of making sure everyone at an exit row meets the criteria & are briefed comes down to the crew, if the crew don't think they meet the criteria, they should be moved, simple. I have noticed, some crew are very relaxed and blase with their overwing briefing's and with the passengers seated there, but what they need to think is "can i trust that person in an emergency? would i feel comfortable relying on them?"
Oz