Its a fairly simple answer. If you took off with an assumed temp set (thrust derate) then you are not at TOGA and the TOGA limit does not apply. If you subsequently set TOGA then the clock starts running.
If you need TOGA for longer than the limit, then use it. There is no point in reducing thrust and hitting a mountain. However the time limit should be taken into account by the company when producing the performance profile for the aircraft.