There is a hard limit for the size of these particular folders, and "unpredictable" behaviour can occur when at or near the limit (2GB IIRC).
This was the case for PSTs and .DBX files until post 2003 where Outlook gained a different database structure for it's PSTs. IIRC the .DBX limit wasn't fixed though. However, it is still good practice not to exceed 1.8Gb in a PST whatever the case.
Background reading:
The .pst file has a different format and folder size limit in Outlook 2007 and in Outlook 2003
Information about the maximum file size of the .dbx files that are used by Outlook Express