I took it to mean that a returning ESTA was one who was not using the very first ESTA for the very first time. I've had many ESTA's (and visas) and I think that this was the first trip to the US on this specific ESTA. That's the queue I went for and it worked for me. I'm sure if I'd joined the wrong line I'd have been sent to the back of another.
The machine prints out a slip with a mug shot on it which you hand in on exiting the immigration hall and before you get to baggage. Normal customs form drills prevail exiting the baggage hall. The immigration officer told me that not all terminals had the machines but I think that these are fixtures at T4-Delta/KLM. I've read nothing about the installation of these bits of kit but it made for a welcome entry instead of the usual 2.5hr shuffle and then the hunt for dumped bags.