Not an Airbus pilot, but I am certain the explanation is same as for Boeings.
20,000 feet is an arbitrary limit.
Below 20,000 feet - flaps speed limits are V speeds. There are such limits.
Above 20,000, should be a Mach number limit, but none are given.
So they just limit flaps operation to 20,000 maximum.
Douglas, to compare, had no altitude limits (DC-8) -
They had flaps VFE 230 KIAS or Mach .46...
xxx
Happy contrails