You don't see any downsides because you are not seeing the bigger picture in what a Perf A aircraft needs to be able to achieve. I guess you never had to suffer a Lawrie Benn 5 day Perf A course back in the day.
Look again at the NTOFP and tell me how bringing the flaps up on speed at an unknown altitude provides you with guaranteed obstacle clearance .