It is not only about personal risk tolerance/attitude to risk, but also about intelligent management/mitigation of risk. I've done IoM, IoW, Scillies, Channel Islands and France several times.
A key consideration for me is sea temperature. I avoid Channel crossings during Winter and Spring, flying only in mid to late summer.
Low temperatures significantly reduce survivability in the water whilst awaiting rescue. Even if you pull your EPIRB in good time, it might take the S&R helicopter or a passing vessel 30-45 minutes to pluck you from the water. In winter that is approaching your survival limit.
I've done the ditching course. Twice. As others have noted, getting into a life raft is not that easy, even from a relatively calm and warm swimming pool.
Over water I fly with a raft and grab bag in easy reach of the exit and wear an EPIRB and life jacket.