"Am I being over cautious?"
No.
I'm in exactly the same position as you; I stayed on the ground today, too. I WANT to regret it, but I made my decision from the local TAFs. It's sunny and blue out the window, but it's also gusty, and those clouds look awfully tall...