It seems to be related to the way the FMC handles numbers internally, 511 equals 1111111 in binary (1+2+4+8+16+32+64+128+256), so 9 "full" bit. Whoever designed it figured that 9 bit would be enough for a distance between waypoints - so the system will not allow anything further apart. If it were to accept it without a change in the underlying code the GC distance would 'roll back', similar to the 2038 problem when the 32bit number counting the seconds since 1970 will overflow on Unix systems. The new 777s probably are only limited by the number of digits (3), so using one more bit (512) will give a theoretical maximum of 1023 which is then artificially limited to 999 for practical purposes. Is that what you were asking for?
