Spending an inordinate amount of time here peering into the terminal window for live load figures as well as the prettier graphical tools. Another poor day.
There is a variable effect in action which isn't a function of straightforward viewing figures and it spans tens of hours or even days.
We went through something akin to this about 5 years ago with the site turning to glue a few times a day. It turned out that the IT bods at the major airframer that doesn't begin with B had noted all the calls out to us and to improve the service to their punters were sucking the whole thing into their server caches

It had us baffled for months.
Danny believes that somewhere in the bb code for the board there is something ugly lurking - for example the search function - that is triggering a far more complex series of sql demands on the processor and database than there needs to be.
I've stood next to him and watched the same load and CPU read outs - with 600 on the board idle readings are negligable - 2 or 3% at most. Where I differ is that I have an utterly unqualified but strong gut feeling that somewhere within the server farm or in a router very close there is a choke point that has been around for a long time and isn't consistent.
As to the server farm it is simple economics regarding location. We get rounds of quotes in the UK each year. Put simply anyone needing our level of server, 24/7 tech support, rigorous back up regime as well as raid plus up to
400gigs of bandwidth per month is corporate, pure corporate and the pricing levels reflect this. However, we ain't corporate. No office, no staff but created and run entirely from our homes, hotel rooms and internet cafes.
We'd love a UK based server. Life would become far simpler for us both day to day and for upgrades/maintenance etc.
We really appreciate the offer of help, I'll be pointing out this thread to Danny so we can have a chat.
Regards from the Towers
Rob Lloyd