Novell did the original deal with the Santa Cruz Operation, who were eventually bought out by Caldera. Caldera then changed its name to SCO (a name deliberately chosen to resemble that of the Santa Cruz Operation) as Darl McBride, SCO's MD had (or was slipped) an idea...
Darl was warned beforehand that the status of the Unix copyrights generally (as well as in particular) was dubious.
An independent expert he appointed to look into the question of Unix code in Linux concluded that there wasn't any (and was ignored).
SCO may have thought that IBM would buy them out (or at least pay them to go away) - a strange misjudgment, knowing IBM's history.
SCO may have though that Linux companies would nervously react to the threats and buy a SCOsource indemnity (as EVI and almost no-one else did).
Microsoft certainly pointed Baystar in SCO's direction (to the tune of $50M) and various monies of uncertain provenance washed SCO's way.
SCO were greedy alright but perhaps they were the patsy all along?
No-one with any knowledge of the copyrights and contracts thought they could possibly win in the end, but the litigation dragged on for seven long years (thanks to some admirable shyster lawyering by Boies Schiller), costing Novell, IBM, AutoZone and Red Hat a mort of money. And of course, casting a persistent shadow of doubt on the legality of Linux, so reducing its uptake by businesses.
SCO lost in the end, as we all knew they would, but someone (no names, no pack-drill) in fact did very nicely out of this mess (and no, I'm not talking about the lawyers).
So for the deep pockets behind it all, it was well worth it at twice the price....
Mac