It's because the whole process is based upon certain traffic flows (CC inbounds for example) being treated in a standard way, and as part of the planned workload of a particular sector.
Tactically, and workload wise there might not have been anything at all stopping you from staying at FL400, however as soon as the system goes 'non standard' it raises many issues and possible problems, both workload and human factor based. We have SOP's, because it standardises things, allows operation to be 'normal' and predictable, allows colleagues to know what to expect and be able to monitor etc. All of these reduce workload. The ATC SA's serve exactly the same function - to stray can break down many taken for granted HF functions.
As a pilot, at times I think we all find some of these restrictions a 'pain' and wastefull. However the key is in the capacity the whole system can offer, and this is by far at it's greatest when our profiles are standardised. The whole thing is one big sausage machine, a very efficient and effective one at that.
I'm sure an ATCO from the 'Westend' will be able to help you more than I on the matter though - hopefully one will contribute.....
30W