Expanding on Mark1234's comment about 'spending longer pushing the headwind', you can see any wind will lose you time by assuming the opposite and looking for the contradiction.
For your 60nm round trip at TAS 120kts it will take you 60 minutes in no-wind conditions. If the time lost upwind was recovered downwind you would also make the trip in 60 mins with a wind of 20kt.
Looking at 60 min trips in general, the air moves 20 kts downwind during them, and you travel through the air at 2nm/min.
A simple 30mins out, 30 mins back takes you back to the same bit of air you took off from. A balloon launching as you took off would be there to meet you after 60 mins, and both of you would be 20 nm downwind of the airfield.
If you want to return to the airfield, you must travel (net) upwind 20 nm during your 60 minute trip. 20nm at 2nm/min is 10 mins, so the new plan is to fly upwind for 35 mins, then downwind for 25 mins. So you spend 60 mins, and get back to the airfield. So far so good.
But how far do you travel during each leg?
Upwind you fly 35 mins at 2 nm/min, which is 70nm, but the 20kt wind blows you back for more than half an hour, so you don't make the 60nm.
Downwind you fly 25 mins at 2nm/min, which is 50nm, but the 20kt wind helps you for less than half an hour, so you don't make the 60nm.
In each case you fall short by the distance travelled in 5 min at 20kt. So there is the contradiction, and so you do need extra time to fly the legs in windy conditions.