Handheld radios
They are not illegal to own. Using them is a bit more of a grey area.
The CAA stopped "approving" handheld radios back in the days of the Icom A3, so any handheld radios purchased since then have been unapproved. The only way to get an approved radio is to buy a (by now) very old Icom A3 or something of a similar vintage.
Back in the real world there are hundreds of unapproved radios in use every day in microlights, balloons and permit aircraft. I've never heard of anyone being prosecuted or even reprimanded for such use - it seems that as long as the radio doesn't cause any interference and is used in a sensible manner nobody seems to mind too much.
I know of one person who tried to get a radio licence for their aircraft to use an "unapproved" radio - the slightly surreal conclusion from the application was that as long as the radio was "CE" marked then it could be licenced as long as it was only used while on the ground ......
As patowalker says, if you are only using it for emergencies (and a radio failure would almost certainly count) then there's no problem. And, of course, there would only be a problem if someone cared enough to take any notice of you using it ....
This is yet another example of our illustrious regulatory bodies making rules which are impossible to obey!