"..The modem connector is a 6 line telephone wire plug, so definitively 3 of the 9 pins on the other end are unused..."
Sorry Vols, I'm now completely confused about your question
"...really good operating systems..."
Do you mean Digital Research's 'Concurrent CP/M' for the 8086/8088 or MP/M II running on a Z80 (eg. TRS-80)? Both sorta work if you're happy with a glass Teletype. Or do you mean DR's 'Concurrent DOS' (needed a 386)? Perhaps you mean Minix on a 8088.
The 286 (deliberately) didn't have the LOADALL instruction for switching into real mode, so the small vs. large memory model problem remained and multi-tasking multi-user systems never developed further for this CPU.
IMS's REAL/32 is the descendent of DR's Multiuser DOS and still being marketed for special purposes (like POS terminals) thought it needs at least a 386.