Odds are that you didn't actually move anywhere during those 4 seconds of take off, it takes that long or longer to start rolling, you probably just heard the engines spool up felt a bit of vibration.
At 3 miles per hour, you move about 4 feet per second, at 5 miles an hour, 7 feet per second, and when first starting the takeoff roll you don't get past 3 to 5 miles an hour for several seconds, so even if you were actually rolling for 4 seconds, you might have gone all of 30 feet (10 meters).
Odds are that runway was more than 6000 feet long (2000 meters), so a 4 second roll would not require turning back.
in 'bad' weather (foggy), an airplane that can't see the runway soon enough will 'go-around', the controller is supposed to keep a large area of the sky cleared for this go-around, so if the plane had landed on the runway, it would be safe to take off, but with it doing a go-around it's no longer safe to take off.
That plane was not landing on the same runway as the one you were taking off from, guaranteed, but now there's another plane in the same area that the one you're on wants to use...