Most experienced pilots will have had near misses involving poor visibility and solid objects and got lucky. Hopefully most will have learned from the experience.
It's possible that the final outcome of this investigation will involve distraction / loss of spatial awareness. It occurred to me as well that the pilot in this case may have been misled by certain visual cues.
Most of us ( me anyway) would expect a construction crane to have a vertical support and a horizontal gib just above the building under construction. This one appears to have had a gib which extended upwards at an angle. The pilot may have seen the bulk of the building and the vertical support and assumed it safe to fly above them, not seeing the gib in the poor visibility. It probably wouldn't matter if the gib was lit as it wouldn't be seen in the cloud anyway. He probably thought that he was safe as he could still see the ground albeit from the cloud base.
I remember an experience from years ago flying along the Tyne. As a crew we were quite alarmed to see a large wire suspended on two quite large towers ahead of us directly on our flight path (in good visibility). This is with at least two of us checking the map! Human nature will get you every time. A single pilot in poor conditions, probably under a bit of pressure is in bad shape in these circumstances.
Very sad.