The problem is that diversions often involve technical problems which ground the aircraft, or the crew run out of duty time. Either way you can end up in a hotel for 12 hours before setting off again.
A simple diversion due to fog on arrival, divert and refuel with everyone remaining onboard with the doors closed, and come back 2 hours later shouldn’t be an issue.