My threshold for taking control away from the flying pilot has been either "do I feel my life is now in imminent danger"? Or "We are now at the point where with my experience (as appropriate to the aircraft type, and nature of flight, of course) tells me that I could not make a safe landing out of this, so I am not confident the flying pilot could either".
None of the times I have taken control have resulted in drama of any sort. I have found that the less said after the fact, the better.
A few times, I have recognized an impending bad situation early enough to be able to say, "would you like some help?" or "would you like me to fly for a while?". In every case, this has resulted in either the pilot flying happily letting me take over, or otherwise recognizing that a bad situation could be close, and taking appropriate action.
A few other times, I have left things too long out of blind faith (which went un-rewarded), and it has simply been "I've got it!!!". Every time I have done this, the outcome has been favourable, and every time, I have sensed relief on the part of the pilot - not anger. Again, the less said after the fact, the better....
Then there was the one time, right seat in a Piper Cheyenne to a very experienced pilot, with a load of people in the back... I just grabbed the controls, and rolled the plane away from the hawk right in front of us. My surprized pilot agreed that my dramatic action was appropriate under the circumstances (he never saw the hawk).
Just as often during checkouts, I have had the pilot try to give me control. I remind him/her, that they are doing fine, and keep flying - I don't need to fly. I find that these pilots generally fly quite well, once they reach a level of comfort.
On the other side of things, I have learned that to not have the other pilot take control away form me, any maneuver which is unusual will be well briefed in advance, and changes in conditions noted with verbal comment. Where I plan a maneuver right through (spin, for example), I explain in detail what I am going to do, and state that I will fly the plane right through. With a compotent pilot, and your demonstration of compotence to that pilot's satisfaction, that will probably assure harmony.
I once was flying a very nervous type flying instructor back from a delivery of a second aircraft. I briefed a STOL takeoff, and that was what I did. I guess he was not expecting this, or what was involved. Prior to the moment, when he reached for the controls to take control away from me, (with nothing said in advance), he apparently had not noticed that my plane has the controls removed from the passenger side. His two hands passed right through the cockpit space, which otherwise would be occupied by the control wheel, kept right on going, and firmly grabbed under his seat. By then we were safely airborne, and climbing out. He later complianed to his boss that I was an "idiot" and "flying around below stall speed". His boss asked him: "If Jim was flying around below stall speed, how come the plane did not stall?" Nervous instructor had no answer for that.
Different pilots come with different levels of comfort. We all have to get along, but some pilots should probably approach some types of flying with great caution, and a willingness to surrender control.