It's probably worth noting that the 737-500/600 are basically the same size as the 737-200 - which was technically a stretch of the 737-100 (granted not many of the -100 variant were built). What killed the 757 was that when the 737NG came along (particularly the -900 variant), it could do most of what the 757 could do and had better operating economics, only falling short on range. Further, the 737 was
way cheaper to build than the 757
'Shrinking' an existing design seldom goes well - as noted the wing is too big and heavy and much of the rest of the structure is similarly sub-optimal. Off hand, I can only think of the A319 as a reasonably successful 'shrink', while the A318 was a major flop.
A great performer, but commercially it fared even worse than the short 737s - there's no way that selling 44 aircraft got anywhere close to covering the NRCs.
The 747SP filled a specific need for more range - trading payload for fuel. I don't think the SP was a financial failure - at least not a major one. The market and NRC structures were far different in those days so it was much cheaper to bring the SP to market than it would be today, and the margins that aircraft sold for were much larger.