brockenspectre,
there are, indeed, regulations (e.g. FOM or G/B OM), which “should” have prevented the scenario you described. The number and size of “live animals in cabin” are limited by aspect of the passengers safety and comfort, and cabin-size. But, most manuals use the word “should”, and, following in smallprint “...these limitations may be varied on commander’s discretion...”, for obvious reasons...
I would guess that what you saw WAS legal, although I personally wouldn’t stick my neck out a tenth of an inch for a dog, no matter how small; which might not be easy at times (celebs etc.). But “Commander’s discretion” also means taking full responsibility for any consequences.
[For example: If you notice a dog in your vicinity, telling the flight attendant that you suffer from severe dog(hair)-allergy and are in no way hesitant to sue the company...- that will definitely do the trick]