Downwash induced drag is real but I dont believe it is nearly enough to compensate for the weight lost. We sling with our Astar with the right side doors off 24/7 365, for visibility more than weight.
As someone else said, if that small amount of weight is the difference between you being able to pick a load or not, you likely need a medium.