what I cannot understand is anybody then choosing to fly in IMC in these circumstances
Nigel, you need to do some GA flying
In the UK, the easiest way to get about is to stick with Class G. That way, you cannot be refused a transit.
Due diligence means you should obtain a RIS when you can but often you cannot.
Combine that with flying at/above the MSA (avoidance of scud running i.e. CFITs which kill many orders of magnitude more people than mid-airs) and you often end up in IMC.
Objectively, it isn't a risky thing to do because anytime the UK weather is naff GA is very thin on the ground, and GA in IMC is even thinner.
Say I fly from Lydd to St Mawgan. I will try to stick to FL040 and that means I need a transit of Solent. Usually I get this but sometimes not till too late, so I have to dogleg under the airspace at 1900ft.
IMC or no IMC.
Anyway, FL040 is probably in cloud too.
FL060 is not possible due to Class A, and a FL100-FL180 Eurocontrol route needs an IR as well as being a bit of hassle to prepare, and it is guaranteed to place you into icing conditions IF there is IMC at airway levels.
But I am sure you know all this.
Thanks to the "make the IR really hard to keep undesirable characters out of the airlines" policy of past decades, very few private pilots have an IR so cannot get the additional protection of controlled airspace enroute.
And training in real IMC is much better than with foggles.