While the 500' rule applies to people but not animals, you can (apparently) get done for causing distress to the animals by flying below 500 feet. Can't remember exactly why - I don't think it is part of the ANO - but I believe it has happened. It's in my notes at home.
T_M: Read the warning on page 24 of Thom book 2 first.....