Once the aircraft is well through the critical drag rise and wholly supersonic, sweepback does little for drag and as hawk says, merely increases the 'wetted' area. Also, as the mach 'cone' begins to play on the leading edge of a swept wing at the higher Machs, the advantage of a subsonic wing flow are degraded. This is why a lot of high Mach aircraft have 'straight' wings, accepting the greater drag rise trans and low sonic for the benefits at higher Mach, remembering also that these a/c have significant thrust excesses anyway and are not designed to spend much time in the critical drag rise regime.