I would say starboard drift:
If you are moving to an area of decreasing temperature, then you must be moving from an area of warm air to cold, i.e. you are passing through the warm front. The jet stream would be blowing north/south, therefore if you were heading east, you would be pushed south, i.e. to Starboard. If heading west, to port.
Actually come to think of it, there's no way you would know, so perhaps it is that your groundspeed would increase...