Continuity of instruction is possibly the most important aspect of basic flying training. Both in terms of instructor and aircraft. And airfield come to that.
In order to focus on the new tasks you need those that you've already mastered to stay constant and therefore non-distracting. Chopping and changing instructor will lead to poor training every time. Its the biggest advantage of an Integrated course. I've been an instructor at both a small PPL school where instructors changed all the time and a large Integrated school where I would solely and exclusively teach a student every single hour of a complete year long Integrated CPL/IR Multi.
When the product is all yours you really care from day one lesson one in how they are taught. When this is just another student for 5 hours of circuits or nav or general handling then - you do your bit but don't really care too much. You'll also be very conservative in sending them off for solos or putting them up for test.
You need to sort your problem out ASAP as you've already suffered badly. I'd ask that the CFI becomes your instructor or you walk and you publish the schools name here. As long as its all true you have nothing to fear.
4 instructors in 20 hours is a joke. From now on the CFI is your instructor. You deserve nothing less by means of reparation.
WWW