SaturnV
There are a lot of variables that might explain the Brazilian military failing to spot bodies or much of the floating wreckage in the search grids that were flown June 2-5, but it would be a pretty incompetent search if something as large as the VS was missed in the area searched on those days.
I think you have "hit the nail on the head" - as much of the info I have seen was related to grids prepared for searching, but no confirmation that they had ever been flown. This I believe was due to miscellaneous flotsum and so-called oil slicks being sighted well to the SE of the Last Known Position. The "focus" of the search became "blurred".
With regard to individual items, I will give you an example of the Outer Spoiler located about 42NM to the N by E of TASIL on 13 June. Between the time it was first sighted and its actual recovery, it drifted to the WSW at a speed of 0.5m/s (1 knot).
mm43