Flying cross channel, I'm okay with an indistinct horizon, so long as I can see the sea on either side, and keep the wings level through peripheral vision.
If I met VFR conditions as described by IO540, I would turn back swiftly, since I don't have the instrument skills necessary to manage the situation. I have a night rating (so not freaked by flying without the full visual picture), but no IMCR.
QDM, I'd give it a try and if you are uncomfortable, turn back into the conditions that were acceptable before.
IMHO you are spot on in considering the risks and working towards a rational decision.