I remember having a similar conversation with my instructor in the early days.
His explanation was that the "setting to neutral" wasn't so much to actually set it to "neutral" but more to make sure that the trim was not set to max forwards or backwards which would cause a bit of surprise when rotation commenced.