flyingpom, that's not a silly question, as the answer is rather counter-intuitive.
In math terms, the reason is because the relationship between time and speed is non-linear:
t = d/s (t = time; d = distance; s = speed).
For a given distance d, if you plot the time that it takes to fly it versus the ground speed, what you get is a hyperbola (x = speed; y = time; never mind the units nor the bottom-left quadrant):
Starting from any point on the curve (e.g. the one corresponding to 120 kt, wherever that is on the diagram) and moving 20 kt to the left and right, you'll see that the corresponding change in time (i.e. the how much you move on the vertical axis, in absolute terms) is bigger when you move 20 kt to the left (headwind) than when you do the same to the right (tailwind), because the slope of the curve always gets steeper going from right to left.