Reacting to the movement over the ground is much less effective than reacting to a change in the hover attitude - hence people who can hover well make small, gentle cyclic movements in reaction to seeing the aircraft attitude change away from the hover one.
Which is why Vertical Reference is a steep learning curve, when you first start lessons - basically learning to hover again from scratch!