And 'air' is invisible also, yet it has killed a lot of people through turbulence, wake turbulence, microburst and wind shear. Just becasue you cannot see it that does not mean it doesn't exist.
The ash particles cause damage even when they are microscopic and invisble to the naked eye. Volcanic dirt/dust also varies in its density and even texture depending on what part of the Earth the volcano is located. In other words, the same shoe doesn't fit all.
As
The The said
In aviation, the safest option is to be risk averse!