Not a 737 or an A320, but my ship has one big difference in single pack ops when comparing QRH to MEL. Applying the MEL requires an engineer to go to the tail compartment and manually secure the pack valve to CLOSED.
The system is therefore in a different state under MEL than in QRH.
The time to descend is a red herring. The higher N2 in single-pack ops is NOT going to make an appreciable difference to RoD in an Mmo/Vmo, airbrake out, emergency descent.