Why was the NG launched? To compete with the A320, which was taking sales from the 737 in both North America and Europe.
The Classic already had CFM56 engines, so fitting an updated model was no big deal. 61-inch fan on the NG compared to 60 on the Classic.
The big difference was the brand new wing, using aerodynamics from the 777. It's more efficient than the A320's. Plus, as said above, the 737 fuselage is "efficient", with a better ratio of seats to external area than the A320.
The excellent new wing allowed the launch of the -800, a 3m stretch on the -400. The resulting seat-mile costs measured up to the A320's and the rest is history.
More interesting is what was going on in the late 90s/early 2000s. Boeing was developing Project Yellowstone, consisting of the Y1, Y2, and Y3. The Y2 launched as the 787 and the Y1 was intended to replace the 737NG with a plastic jet using 787 tech. But single-aisle sales grew beyond what's possible to mass-produce in CFRP. Then Airbus just fitted next-gen engines to the A320 rather than develop a new, better airframe, knowing that Boeing couldn't do the same. Boeing tried anyway and the rest is... 🙁.