For all your aviation maths, go here to view the Aviation Formulary:
Aviation Formulary V1.44
Distance to horizon
At a height h above the ground, the distance to the horizon d, is given by:
d = sqrt(2*R*h/b)
b = 0.8279 is a factor that accounts for atmospheric refraction and depends on the atmospheric temperature lapse rate, which is taken to be standard.
R is the radius of the earth. Note that the earth is assumed smooth- likely only true over the oceans!
For h in feet and d in nm:
d =1.17*sqrt(h)
i.e. from 10000 feet, the horizon is 117nm away
(Reference Bowditch American Practical Navigator (1995) Table 12.)