Boeing recommends many things. However I personally disagree with the all on or all off philosophy.
whats the problem with deselecting spd mode with A/T armed?
why would leaving it armed cause confusion. if it confuses you then you shouldn't be flying it.
I can see only good points about it.. underspeed protection, automatic goaround thrust selection when toga is pressed.