Think of your car. If you take your foot off the accelerator whilst going up a hill, it slows down quickly. If you do it on the flat is slows down, but not so quickly. If you do it on a slight down hill, it slows down, but hardly at all. If you do it on a steep down hill, it does not slow down, and will even speed up. The principles which cause these things to happen in your car, are exactly the same ones which govern whether a plane can slow down rapidly or not. (ie gravity, and drag). In your car, you can increase drag by using the brakes. In a plane, we can use the speed brakes, and also the flaps. The problem is that the heavier the aircraft is, the smaller the deceleration resulting from these devices is. And so it is much easier to slow down on the flat, and then descend, rather than to try to decelerate whilst going down hill.
The physics of it all, is basically the conservation of energy. Potential energy (energy due to height), must go somewhere as you descend. This energy is converted into kinetic energy (energy related to velocity), and also wasted due to drag. If your rate of loss of potential energy (ie rate of decent), is the same as the rate at which energy is lost due to drag, then you will neither accelerate nor decelerate. If you descend quicker than this, there will be some spare energy left over, which is converted to kinetic energy and so you speed up. If you descend slower than this, the drag is taking away too much energy, so some has to come from somewhere else too. It is taken from the kinetic energy, and hence you slow down.