(Altitude^2 + 41849280xAltitude)^0.5 / 6076
The Altitude is much less than 41849280 ft, so the first term is insignificant, and the second dominates. That then leaves Distance/nm = 1.065 x (Altitude/ft)^0.5 which is the correct geometrical answer.
So why is the prefactor for VHF 1.2 rather than 1.06? Radio rays (and light rays) don't travel through the atmosphere in straight lines, but rather they bend, mostly because of the variation in density of air with height. So the line you have as X is actually slightly curved in the same sense as the curvature of the earth. That adds about 10% to the answer.