Sorry to be technical, but a thorough understanding of how the automation systems work and interact is important.
The autothrust does not "control" the speed. When active, the autothrust adjusts the thrust to achieve the desired speed as ordered by signals from the FMGS / FMGEC, which is controlling the speed. When in Managed Speed mode, the FMGS determines the appropriate speed depending on the active PERF mode, thrust lever position, cost index, aircraft weight and flap / slat setting. In Selected Speed mode, the FMGS takes orders from the pilot's selections on the FCU and sends those orders to the autothrust to achieve the desired airspeed as "controlled" by the pilot.
Also, after takeoff, F and S speed are not just reference speeds, they are minimum safe flap / slat retraction speeds, i.e. they provide the safe airspeed margin for retraction of flaps and slats. Part of the "logic" of the system is that the FMGS will maintain S or F when in Managed Speed in the APPR phase so that if the crew elects to retract a previously selected flap setting (change of runways, additional maneuvering required, etc.), it would be safe to do so immediately as S or F is already being maintained.