The captain may have been "managing expectations" and telling you the worse case scenario, whilst hoping for a slot improvement.
TBH unless some clearly catastrophic is happening en-route/at destination in situations such as this the crew are almost always best placed getting the doors closed and telling ATC they are ready, just in case a early slot appears, rather than having the passengers disperse around the terminal.
I note you ask about returning to the gate - did you remote hold (i.e; taxi then stop somewhere on the airport) prior to departure? Sometimes the gate you are departing from is needed for a scheduled inbound aircraft , so you taxi on schedule, then remote hold awaiting your slot..that can happen with many airlines, especially at busy airports.