If there were no referees, football would be a much different game.
You could ask the players to kindly not kick their opponents, but as long as there there were no rules and no referees, many would play football more like rugby, or worse. The game itself would be degraded.
But if you set the rules and enforce them, football is not degraded. It is not perfect, but if works reasonable well and we can enjoy it.