There was nothing major preventing Boeing building a 747 derivative with a full-length upper deck. The 747-500X and -600X were crebible, but would however have been a very major change, but possibly not that much greater than what the 747-8 actually became.
However, at the time it was judged as viable only if Airbus did not build the A380, and not even needed if they didn't. Boeing decided to play poker and Airbus decided to build the aircraft. Quite why they thought the "cheap" substitute (-8i) might make the grade in the presence of the A380 remains a mystery.