I don't consider 20 minutes a long time at all. I'd keep them both running. Having the PTU or yellow pump running is incredibly annoying from a passenger perspective, particularly for those seated nearest to those pumps (A320).
Obviously for a 90 minute taxi at JFK, I'd consider shutting one down, but again, the noise in the cabin for that length of time may give me pause. If I know the taxi will be that long, I'll request extra fuel. We may be too heavy to make a SE taxi practical anyway.