I think of it this way:
Drag has two components, induced and parasitic. At the same IAS, parasitic drag is (about) equal in each (heavy vs. light) case.
But heavier weights, there is more potential energy to be disposed of by parasitic drag. Induced drag is greater, and thus parasitic drag is a smaller fraction of the total. Ergo it takes longer to burn off the potential energy (i.e. descend).
Not strictly accurate I know, but the major factors are there.