EU-OPS has also been quoted several times and you have completely ignored it.
The procedure for an engine failure at 500 feet is to continue the climb to acceleration altitude in the takeoff configuration, reducing the speed to V2 +10 if you are already above that speed. This is specified in the 604 and 605 manuals, but not in the 601.
I'll turn your question around. Where in the AFM does it say we should retract the flaps as soon as aerodynamically possible? You know perfectly well that it doesn't say that anywhere, so your bleating about following the AFM doesn't hold water.
I'm coming to the conclusion that the real problem is the African sun is getting to you...