I know what you mean
The downwash is not due to the vortex. If there is lift there has to be downwash, according to Newton. If there is angle of attack, then the air is pushed down and the wing pushed up.
In an infinite wing there would be downwash but no vortex.
If the wing is finite some air leaks from lower surface to upper surface, thus creating the spanwise flow and the vortex. The less the aspect ratio, the more the spanwise flow, hence the more intense the vortex is. (less aspect ratio means that a larger proportion of the air around the wing will leak and create vortex).