"because if you do have an empty tank on one side, then selecting "both" will draw air from that side, and probably not fuel from the other"
On a gravity fed system - I don't think so !
But flying out of balance may cause a fuel feed line to become uncovered.