Since drag has an upward not downward component it cannot make a glider go down. Flying at more or less than min sink decreases the upward resultant of lift & drag so that glider is affected more by gravity. The bigger the upward component the longer the glider stays aloft, the smaller the upward component the quicker gravity pulls it back to the ground.
As to which glider remains airborne longest (not the same as which goes furthest from a given height), lack of drag is clearly not the only influence. The glider which produces the biggest upward resultant of drag *and* lift is the one that stays up longest. So, drag is only the determining factor when they have the same mass *and* they are both producing the same lift. Possibly we should require they have the same wing loading as well....
(and gliders that stay up well are frequenty not gliders that can go places, especially not upwind places)
If you rephrase to say that drag has a big influence in how long a glider can resist gravity then yes, obviously so - as does lift. But since gravity is the only downward force, gravity is the force making a glider lose height. Unless Newtonian mechanics have stopped working.