Everywhere was promising CAVOK, it's true. But all this means is that the vis is 10km. Going south into a low winter sun it can instantly be half that. And that's before the TAFs start to be even slightly wrong.
I flew with a student from Nottingham to Sywell, and if it hadn't been for the fact that we were following line features, and were in a helicopter so we could slow down and even land if necessary, we might well have turned back. Vis was about 5 - 6 km, and it was hard work. But on the ground - glorious day.
You made the right decision.