tuc,
You have got me doubting whether I should have said 'radar' or 'radio' altimeter, so I shall stick to 'radalt'!
I don't know what beamwidth a rotary-wing aircraft's radalt would typically have, but the scenario you describe would certainly be possible with the radalts fitted to fast jets. The beamwidth of older ones was about 90° to ensure that they stayed locked on at moderate bank and pitch angles, and they returned the "height" reading from the shortest range return. This was noticeable when flying parallel to vertical cliff walls, when it was possible to get a return from the side at small bank angles. To address your specific scenario, someone flying into a hillside in level flight would see constant barometric altitude, while the radalt progressively decreased to zero.
There are some newfangled 'covert' radalts, designed to be better in tight EMCON. Whether these have a narrow beamwidth, electronically steered to the vertical axis, I know not. If they did, it might be possible to fly into a vertical cliff wall while getting a height reading from the valley floor.
Back on topic, TCAS II believes whatever radalt input it gets when deciding whether to screen out "descend" instructions. They didn't account for covert radalts or valley flying when drawing up the spec! The cooperative decision on vertical deconfliction is taken with reference to barometric altitude at 1013.2hPa, as fed to the transponder, so there should be no chance of undulating terrain compromising the integrity of the instructions (only exception being where the 'low' aircraft is forced to climb by terrain, in which case the 'high' aircraft should get an 'increase climb' instruction). It's worth remembering also that GPWS warnings have primacy over TCAS instructions, even in civvie airliner land; rest assured that system designers had actually considered the possibility that vertically-based deconfliction instructions near the ground might carry their own risks...
TCAS II is not ideal for fast jets but it is better than nothing, which following the failure of the bespoke programme (which I had always understood to be much later than 1997; I thought 2004...) was the only alternative. Even in situations where its deconfliction instructions are inhibited, it will cue aircrew to look in the right place when squawking traffic approaches. It probably would have saved 3 lives in 2012. Let's see what the SI has to say over MoD taking 10 years to actually get round to fitting it.....