invertron
They could see the landmass (it was visible from the NI coast at he time) but it was covered in localized (indeed ground hugging) mist which obscured ground detail this would have affected their ability to judge their distance off visually to any degree of accuracy. Further, the cloud base met the hills and so perspective could not have been had by watching the top and their very much end on approach did not give any laterally displaced topographical features for perspective either. These conditions are very common at the Mull owing to the prevailing wind pushing moist air up the slope.
However, this said, the rules for helicopters still allowed them to be visual at their level in these conditions (clear all around except for ahead with the sea below clearly visible) provided there was clear air ahead enough for them to have slowed down if need be so as to remain in clear air. Although waypoint A was very close in, had they started their turn at the geographical location of waypoint A they would have turned in the clear and not have broken any rules. However, this crew in particular had no confidence in the accuracy of the SuperTans system as fitted in HC2s (Doppler is combined with GPS unlike the earlier system as fitted to MK1 Chinooks, which can result in degraded performance over water) and therefore continuing past waypoint A (as given by the SuperTANS) may not indicate control problems but may simply be that they were going to start their turn judged on something else (looking for a familiar feature or using some other method as ZD576 did not have radar, a suggestion is DME).
It would indeed have been unwise to continue at their speed with nothing else to go on but their visual judgment of their distance off that amorphous grey mass and I do not believe that such an experienced and professional crew would have; while they would not have been allowed by the rules to rely on anything else but visual judgment for terrain avoidance, a nice digital readout telling them they still had, say, ½ a mile to go would have put a strong bias on what they thought they were seeing remember, if you are unsure of your visual reference (like temporary disorientation) the reaction is to check the instruments if they had nothing else to go on, they would have turned if they could (and they demonstrated that they had control in the last moments by starting an evasive turn). If the DME had been altered/moved to give a wrong reading it would have been hard to pick this up by judging the distance to the mist (try judging your entry into a cloud bank to better than ½ a mile). As they got too close the shoreline would have been seen to pass under them, they would have entered the mist, and the RADALT alarm would have gone off all about the same time which coincides with their last desperate evasive maneuver.
ZD576s TACAN CU was set to ch 107, the Macrihanish TACAN, but as there would not have been LOS at their level on the last part of the flight they may not have been able to get a DME reading from it. However, the MK2 avionics would have allowed ZD576 to use portable/transportable DMEs had there been one at a convenient spot. This system also gives an approximate (+/- 4deg) bearing to such DMEs and ZD576 did make a small turn RIGHT at waypoint A which could be consistent with heading towards such a beacon located behind the lighthouse (the helipad?) presumably to turn at a given range from it.