I had a very similar experience as a junior skipper on the 320 a long time back but it's still clear in my memory - lucky in my case the runway was long.
One T/R deactivated, F/O flying, CAVOK but dark night/ cockpit. Reminded him at 1,000' of the deactivated T/R - maybe not a good thing - multi-cultural cockpit, low time FO, he reacted (I didn't see) by lifting his hand off that TL. At the Retard call he only closed the one TL, the other remained in the Climb detent. It was only at about the sixth Retard bleat and observing a clearly asymmetric condition that I realized things were amiss, took control, closed the other one and kept it on the paved surface.
Should we have gone around? Retrospectively no - the spool up time on the engine that was retarded would have proven potentially disastrous.
There, but for the grace of God, go I.