Hi Steve,
You're right to an extent but you have cause and effect the wrong way round. To fly slower we have to decrease power and then raise the AOA to maintain the required amount of lift. If we want to go faster we increase power and lower the AOA to maintain the required amount of lift. If all we were to do was raise AOA to slow down, we would indeed slow down, but we would also start to climb. That's why elevators only control speed in constant power scenarios. Climb or decent. So, we primarily use elevators to go up or down, and power to go faster or slower.
Another way to think about it is energy management. If we want to go faster, or up, we need to increase the amount of energy we have in the system. We do this by adding power. If we want to go slower or lower, we have to remove energy from the system. To do this we have to remove power.