From flying gliders in the past the answer is it depends on the wing profile. Some wings get better Lift/Drag ratio with flaps extended, some don't. Some even get the best Lift/Drag ratio with negative flap setting. Assuming you're flying at speed for best Lift/Drag ratio, changing your flap setting without changing your speed will decrease Lift/Drag ratio.
In competition glider flying, some advanced gliders change to a negative flap setting in order to increase the speed for best Lift/Drag ratio, therby being able to fly faster with a lesser loss of altitude.