From your description it looks like you're maintaining cruise power, and a speed well above Vfe, until the end of downwind, and then need to rush to get rid of all this energy. Reduce power to, say, 2000rpm when you join downwind, so that you can apply the first stage flaps not long thereafter. That should get you in a much more comfortable position by the end of downwind.