Actually, it can still be a problem for the crew behind you on the taxiway if you're that close (and depending on the angle of push) because they still have to stand in your wash and disconnect their towbar, mechanics have to stand in your wash to wrap up any preflight concerns with cockpit crew on long chord, and the marshaller gets to stand in your wash while they wait for you to clear out so they can get their a/c out off.
More likely your ground crew was running slow and you got bumped out of order by a faster crew two gates away - your pushing back on top of them is more a congestion problem than a true safety hazard. And as they say on the ground - move it or lose it.
------------------
Patience is letting your motor idle when you feel like stripping your gears.