It can't be out of memory!!
Oh yes it can. This sounds like good old crap coding. It depends what language they're written in, but let's take this as an example:
How it should work:
Application needs memory for something, asks OS for memory.
OS gives application memory.
Application uses memory.
Application gives memory back to OS when it's finished.
How it shouldn't work:
Application needs memory for something, asks OS for memory.
OS gives application memory.
Application uses memory.
Due to sloppy coding, application does not give memory back to OS.
Application needs memory for something, asks OS for memory.
Application uses memory.
Due to sloppy coding, application does not give memory back to OS.
Application needs memory for something, asks OS for memory.
Application uses memory.
Due to sloppy coding, application does not give memory back to OS.
Memory disappears into a black hole somewhere. The OS thinks the application has it, and the application doesn't realise it hasn't given it back to the OS. This is known as a memory leak.
When the error occurs, you could try opening task manager. On the process tab you'll be able to see how much memory the OS has dished out to specific applications. If that doesn't narrow it down, try Sysinternals Process Monitor; it should go pretty mental when such an error arises and point you in the direction of the problem. A start, at least.