A speed increase is usually no problem; burns a bit more fuel, but saves a bit of time.
A speed decrease, especially if the airplane is already on an optimum VNAV descent profile, could cause problems. Many airplanes do not descend and decelerate well, simultaneously. Excess use of speedbrakes and early deployment of flaps and landing gear may be required, causing even more fuel burn, and possible discomfort to those in the back.