Any large buoyant object in the boundary layer is bound to be skittish in a slow sense.
As a simple experiment, the Met Office courses for graduates with 2.1s or Firsts [and the occasional PhD] went into an open field in daylight and launched simultaneously, and tracked, a red balloon, a white balloon and a blue balloon from a short and surveyed base line.
Each was tracked by theodolite. The divergences after a minute were substantial, after 2 minutes were multiples of the baseline. After 5 minutes they were separated by football pich dimensions.
The students then went indoors and were asked what conclusions they came to.
Predictability in the boundary layer is very very poor. Which is part of the reason to employ proper pilots for proper aircraft.