All the seperation required in hold is done with altitude, so there is no real danger of flying into anyone if you are a bit off.
Not always true. The Holds can also have built in separation from other holds close by and also SID tracks. For example, the LANAK hold for Glasgow is very very close to the Edinburgh jet departure SID. If you wander outside the protected airspace then you will be endagering yourself and others.
So, not being able to accurately fly one in IMC could ruin someones day.