@Boyington,
The potential problem is retracting the flaps with ADR OFF - as this drives the RTLU to "Hi speed" position and can damage it.
Extending the flaps with, or without ADR's shouldn't cause any problem, as both actions will just drive the RTLU to "lo speed" position.
P.S. The above is based on what is written in the FCOM and my understanding of the system. Perhaps someone with access to the AMM could confirm it...