Some people are just not suited to instruction, and others may get on like a house on fire with some students, and not at all with others. It is the CFI's job to weed out the misfits and incompetents - he should be flying with all the instructors regularly - and to juggle the personality conflicts to as many people's satisfaction as possible, without pandering to the professional whingers.
Of course, it's very difficult for the CFI to know what's going on in those little cockpits up above the airfield, but an open office door, and a trained ear and eye in the bar at the end of the flying day, can help immensely. Some places even run their own local Confidential Report system, for just this kind of situation.
The bottom line is that, as the customer, you should point out your concerns to the CFI. It is then his responsibility to do something about it, if he feels your complaint is justified, and if he is actually able to do anything. If he can't, or won't, you can take your money elsewhere. But he should let your FI give his side of the story as well; there should be no judgements on just one side's account of the situation.