Perhaps it is because for a higher sweep, the distance the air flows over the wing will be greater, ie straight wing chord / cos (sweep angle). Hence more drag?
Does this question pertain to the case where the wingspan (distance from wingtip to wingtip) remains the same for all sweep angles (ie, the wing would have to be made longer)?
Or does it decrease as the wing gets swept back?
An interesting question....