Maybe it's choking on one particular malformed/corrupt e-mail? Any chance you can download & then delete all the e-mail using another mail client, computer or a browser interface?
It's IMAP not POP3 le Pingouin, therefore protocol only downloads headers by default unless configured otherwise.
BusyB...
For testing purposes, could you perhaps try another mail client such as Thunderbird to see if the problem can be replicated there. At least then we can determine whether it's on your mac or your ISP.