You basically explained it. Until the pitot is clear if the down wash, it's not getting clean air to be able to produce the needed ram-air effect of forward movement.
The down wash can also create a vacuum effect inside the pitot. It's not uncommon in certain helicopters with glass panel avionics to show a negative airspeed until you've past translation.