The guy in the tower is wrong and you are right.
A flashing white to an aircraft in flight is not applicable and therefore that must be the same as no light signal at all.
I would have done exactly what you did, stay in the circuit and wait for either green or red or a combination thereof
