FFF,
The one time I got seriously lost was when I misidentified a ground feature and made a large heading change. It's worth bearing in mind that winds are rarely that far off what they're forecast to be. How different can they be? For instance, a wind of 30 kts at 90 degrees to your track means you need to correct by about 22 degrees; a wind at 45 degrees at 20kts means you need to correct by 10 degrees. If you're 10 degrees out, after 20 miles, you'll be about 3 miles off track, and even in 5km vis still be able to see the town or whatever that should be there. And that's a pretty extreme example to take. The only time when that doesn't apply is if you're correcting in the wrong direction, ie a wind from the right becomes a wind from the left. But even so, it would have been nearly a head/tail wind to start with, so it's unlikely to make that much difference.
So if you find yourself making huge heading corrections and thinking it's because the wind was not as forecast, maybe take another look.