There have been a number of times when I have seen an aircraft below me, looked on TCAS and it has actually been above. When we get closer, the optical illusion disappears. If you saw one of those coming towards you and got a "traffic" warning because you were at TA only, then you may well climb into his path. What's worse, is you both may climb.
If you have a conflict with say a Hawk while both flying VFR, you are unlikely to see it. In fact when the guys at Valley had some war games with some F15's from the states, the hawks won every dogfight, because the F15 pilots couldn't see the Hawks (they are very small). If experienced fast jet pilots can't see them, then having your system set to RA will definitely help.
What happens if you are flying VFR and get a "traffic" warning (while set to TA) from someone below you who is not actually a conflict - you look out of the window and can't see them. What do you do? Manouevre or not? What if you just haven't managed to see them and they ARE a conflict? I would say RA every time. A brief excursion into cloud (20 seconds say) is less dangerous than the scenarios above.
A TCAS climb isn't a full power EGPWS climb, it may well be just a few hundred feet a minute change, or stopping of a level off.