Two considerations ..
(a) use of flaps at high speed will end up with a maintenance cost penalty and this can add up significantly for a routine usage pattern.
(b) the 20000 ft admonition is, if I recall, associated with not having demonstrated some certification requirements and may be associated with undesirable roll rates ... quite apart from the question of why would one want to drop out flap at that sort of level ?