Will,
That has always been my view, but I can imagine that there might be an optimum (oh, alright "sweet spot" ), and if Richard says there is I would be happy to believe him.
With
real 
operating systems you're pretty much right; however it's my understanding that Windows divvies up the memory into fixed-sized chunks / tables (depending on how much physical memory you have) for various purposes, which is why you get those "the system is dangerously low of resources" messages, despite there apparently being shed loads of spare memory -- if one of those fixed-sized tables fills up, you're stuffed
I'm quite prepared to believe that once you get above a certain "sweet-spot" you're into deminishing returns on the amount of effort required to manage the additional memory ...