We do plenty of improved climb TO's where I can assure you the end of the runway looks very close at V1. In fact to the naked eye it looks like it would be impossible to stop, the figures tell otherwise though. Another potential hazard is a runway incursion.
Even if the figures allow a three second delay, this includes recognition, etc. You still will need to lift your feet from the pedals in order to reposition them further up. If you have a significant input on the pedals (due to a crosswind or an engine failure) you may not be ablle to release the pedals even momentarely without veering off the centreline.
I could reply to EL Peligroso likewise that if you think you may drag a brake on TO you might need some further training. It does not enhance the argument however. The problem is not finding the brakes but being able to release the control input to reposition your feet.