Because speed brakes should not be instinctive especially at high altitude when slow (for ATC). Their effect of the min speed is marked and deployment is to be a conscious decision.
And how would the other guy know how much you're extending
And you could hit the rudder by accident...
I'm sure there's more!