You are kept at the gate for your own safety, and so that the boarding process can be completed in a controlled manner to ensure an on-time departure where possible.
Pax are herd animals, and like most herd animals have to be treated as though they are all stupid. Left to their own devices, they will most likely try to board the wrong aircraft (it ain't necessarily the one outside the gate), wander all over the tarmac, try to climb steps not attached to aircraft, climb external stairs on airbridges, walk in front of/under/over ramp equipment etc, etc... Where one goes, the others will follow.
It can be hard work keeping them out of mischief at the best of times, let alone if they were allowed out at a time of their own choosing.
There is also the weight and balance aspect of boarding pax to consider, as some aircraft types like the A321, can be prone to tipping if too many pax trample up the rear steps first. Most pax are extremely unlikey to be aware of that.