My understanding of why Boeing decided to stay on the 737, is that some major players like SW and Ryanair did put a lot of pressure on Boeing NOT to develop a 797, simply to have one type in their fleet, and not two.
That could however be from around the time when Boeing decided to develop the Max, and not when they decided to develop the NG.