Or are you implying that a speed restriction is used to prevent overflying built up areas?
It often is.
Just in the same way that it can be used to prevent you hitting an obstacle during or after a turn.
Remember that the standard speed restrictions that we have to comply with as pilots do not have to be published on the charts. They only need to specify the speeds if they are different from the standard speeds.
There is no need to tell you on the chart what the maximum speed in the hold is or intermediate approach or final approach or circling or on a missed approach or a turning departure because you are expected to know the standard speeds.
ATC can not absolve you from the requirement to comply with the basic speeds appropriate to your category.
If you were on an arrival and ATC say "no ATC speed restriction" or "high speed approved", does that mean you can fly the final approach at 250Kt and expect not to hit any obstacles during a circling manoeuvre started at at 250Kt? No.