Or the trees or water. When recently doing my RPL training the instructor pointed out that you could tell form the water in the nearby river. The calmest side was the side the wind was blowing from, i.e. close to the bank the water is protected from the wind. On the down-wind side, it cops the full force. Amazing how obvious it becomes once you know what to look for.
Remember the wind the GPS shows at 500ft cam be very different from on the ground.