Correct this is is getting interesting!
The reason is he possessed no positive inertia relative to the surface he was standing on. But you did.
No, the reason is he has enough inertia to resist your momentum.
Now visualise the same 20 stone guy heading for the touch down line at 15 knots. You are standing still this time. To throw him off balance you'd only need a gentle shove as he sped by ... 20 knot wind or no wind, the above example is sound.
Now he still has inertia, but plenty of momentum, which you can "vector" with a glancing blow, maybe deflect him into touch. You would feel this inertia well enough if you stood directly in front of him. This inertia is why you can't stop him instantly, or even push him sideways at 90 degrees (unless you are bleedin' enormous, i.e., have much more inertia than him)
Inertia is resistance to change of velocity/direction, and is present regardless of whether moving or stationary.