Could be nothing at all to do with your own set up and connection speed - more likely it's you ISP's mail server that is causing the delay -
Check your ISP's web site for any info on service availability, or e-mail their Customer Support department to query the problem - it may be that they are upgrading their system / carrying out maintainenance etc. which can result in delays like this.
NTL UK had the very same a couple of months ago due to maintenance work - seems better now !