With the drysuit do you actually fly in it? I can't really swim more than 10m and I too thought the cold would be the issue and thought without a liferaft you are going to be cold - so developed a view that you either do all or nothing.
Yes, you are supposed to wear the drysuit while flying. In fact, it's a good idea to actually have it fully zipped up. Most, if not all, are made from Goretex or something similar so it's not too bad.
The first time I tried my drysuit on, it took me about twenty minutes to put on, as there's a certain knack to getting your head through the seal. And this was in a comfy living room, with help. In a GA aircraft, while dealing with an emergency, there is no way I'd be able to put it on in time. And a partly-on, or not zipped up drysuit is actually more dangerous than no drysuit at all: It doesn't do anything for insulation but greatly restricts your movement (and thus, for instance, your ability to enter a life raft) because it fills with water.
You are right that the cold is the major issue here. So underneath the dry suit you have to wear a few layers of clothing that provide insulation. Divers use special "bear suits" which are coveralls made of fleece or something similar, but for aviation emergency use you can probably get by with normal pants (jeans possibly), a t-shirt and a sweater or something like that. The main issue is that you stay dry, because air is a much better insulator than water.
As far as life rafts are concerned, remember that a life raft without a dry suit is only partly solving the hypothermia problem. You will very likely get wet before you can get into the life raft. And in the very lucky circumstance that you can step off the wing straight into the life raft, you'll probably find the life raft floor is wet from the deployment or spray. So you will have to expect to get wet in any case. Without a drysuit, that means your insulation is now gone.
Best solution is obviously a dry suit plus a life raft. Plus a life jacket in case you do get into the water. But if you have a limited budget and no training in working with life rafts (deploying them properly, turning them over if inverted, getting on board, deploying the canopy & water ballast, ...), I'd say the dry suit plus a life jacket, but minus the life raft, is probably the next best solution.
Not being able to swim 10m is not a problem at all, by the way. Where would you swim to anyway? The only thing that would conceivably be within easy swim distance of your ditched aircraft would be the life raft. And if you deploy it properly, then it's either tethered to the aircraft, or to you. So you can simply pull it towards you, or get into the water and pull yourself towards it. Other than that, conserve your energy and don't swim anywhere.