Simple answer is that you don't have to pull them down. It may be courteous, but you don't HAVE to, doesn't matter what any FA says.
Most airlines provide eye shades, if a pax is trying to sleep and the light is affecting them, then they should put the shades on .... simple really.