A few considerations -
(a) FOD has been mentioned
(b) circulation and reingestion
(c) aircraft shake, rattle, and roll is not really a problem
(d) the acceleration question is not a major consideration as the acceleration peak is reached quickly regardless
(e) distance penalties are not a major consideration if the start commences from a rolling taxi and the thrust is set promptly
(f) distance consideration is relevant for a very light aircraft on a short runway as the breakaway thrust is low and the acceleration at lower thrust levels is not insignificant. In the case of a heavy weight takeoff, by the time the thrust is set, we haven't ventured very far down the runway