With my old airline it was easier, you couldn't "pre-request" these seats, and we only ever had to move one guy, who needed an extention belt. He was good about it though, which was lucky - that kind of situation can be mortifying for both the passenger and the crew member.
I can't say how I would have handled it if he refused to move, after we asked and pointed out the rules to him, I think you probably did the right thing, but explaining that the flight was going to be delayed - no one really wants to be THE ONE on the plane that caused it to be late, I'm sure!