Question is what should I have done?
Two things:
1) Make sure you know who you're following and where they are. If you are unsure, ask. If you are assigned no. 4, you must follow no. 3. If you can't easily arrange your flight to do that, tell ATC.
2) Follow the one you're meant to be following!
Mix-ups of the sort you describe do happen in the circuit -- it can be difficult to judge distances and speeds, even for ATC. In my experience, the higher the calibre of the other pilots involved, the more tolerant and accommodating they are.