The main bugbears with weather diversions for CBs, heavy showers etc, is unplanned ( and unplannable) navigation changes, and resultant frequency changes etc, and controlled airspace incursion (possibly inadvertent, with all the knock-on repercussions of that)
In short, they can really cock up your whole day!