2) They increase the stall speed and make Vref no longer valid as a guide to 'safe' margins
Monsieur Airbus kindly adjusts the minimum speed (Vls) on the speed tape to take account of the speedbrake. He will even add thrust to stay above Vls if necessary, with autothrust engaged, whilst of course sternly rebuking you for power against speedbrake with an amber caption.
Speedbrake + flap is allowed on the A320 except for flap full. If it's allowed and necessary, not using it is a poor choice. It's a flight control, use it as such. Whatever is necessary (within SOP's) to be stabilised by 1000/500' should be done.