Not using the speed brakes when necessary is equivalent to not using the foot brake in your car. You can manage without it sometimes, but other times you need to use it - especially since you don't always know what ATC's plan and traffic flow situation is. No problem.