F-104 wing is not that much different in plan from the F-22, except for the tip chord as a % of root chord.
The vertical plane of max thickness is nearly straight (i.e. little sweepback) in both cases. Both present a need for more aggressive area rule in the fuselage.
And Mach would seem to influence the planform much more than altitude, don't you think? Altitude influences the need for lower wing loading, thus (for a given GW) the required wing area.