Is there any way round this, other than turning the speakers off, and using headphones to receive the other side's conversation?
I generally use headphones with built in mic for exactly this reason. As you say, it takes time for the conversation to travel over the internet - you can't do anything to change that.
But try turning the speaker volume down at your end to the minimum that you need to hear what's going on, and keep the microphone well away from the speakers - and ask the person at the other end to do the same. You might find that sorts it out.
RC