I am not sure if it says somewhere you *have* to do a downwind entry, but the AIM certainly *recommends* you to do so. If the A/FD would give specific instruction on entry, I guess that would be a rule to go by.
Advisory Circular:
AC 90-66A Recommended Standards Traffic Patterns for Aeronautical Operations at Airports without Operating Control Towers