With spoilers deployed, the airflow separates at a lower AoA (in those sections where there are spoilers deployed).
Not the whole wing, but if you are extending speedbrakes you can have a significant part of the wing in a stalled condition at speeds well above clean VS1G. Those stalled sections will give control problems and the margin to a totally unacceptable situation is reduced. I guess that is why the VLS increases with the speedbrakes extended.
There was an incident of a 320 where a spoiler remained deflected due to a mechanical failure. The VLS remained the same, but the AP had some trouble maintaining lateral control during approach, when speed was VAPP. If I recall correctly. It was that small section of the wing that was creating the problem, probably.