Not current on the 320 but from memory, when you activate approach phase the box memorises the weight at that point, so if you do it too early the speeds will be in error as the weight reduces further.
I had either forgotten this, or never knew it. Our lot tell us to do it during the descent "in case we forget"* but you actually only need to do it in order to slow down and start getting flap out, so that could be at around 20-25 miles. If you are flying in Managed NAV, the system will automatically activate approach itself.
Cheers !
* - and it is VERY embarrassing to be flying in selected speed on the ILS and having forgotten to activate the approach, you then push for managed speed at about 5 miles, only to have the A/THR put in a shed-load of power to get you back to 250kts !!!