If you hit TOGA at T/O, and nothing happens, then the TOGA function doesn't work, right. In your case you're simply operating the thrust levers manually. No A/T, no TOGA and no RETARD.
More importantly;
after hitting TOGA the A/T+FMC will command T/O N1, but if you reach 84 kts before T/O N1 is set, A/T will command THR HOLD at whatever N1 that has been reached. You'll then have to increase thrust manually.