Why would it be unsafe? If you are thinking of fire, you are far more at risk filing your car up because it is open line.
That said, why would they keep you waiting on the steps? I've been on board many a time whilst the aircraft was refuelled.
Thinking more though, I assume that they didn't pull the steps slightly away from the aircraft (passengers might fall through the gap). If they didn't, there is the danger of damage to the aircraft against the steps, due to the aircraft dropping with all the additional weight of the fuel on board.