The population density of a stadium is a lot greater than some random part of central London.
It comes down to a numbers game. You shoot the aircraft down before it reaches the stadium and perhaps 2,000-people are killed/injured. If it hits the stadium then you're looking at 20,000-people.
That said, the people in the stadium are choosing to be there despite knowing it is a potential target...