The other islands are tiny, and apart from getting submerged by tropical storms from time to time, there is no fresh water. Infrastructure wise there are no buildings, no harbours, no airstrips, and no electricity.
I suspect that people who have never been there believe that they are just like an undeveloped Maldives, that might work for a very few visitors on Diego Garcia itself - if the military left, leaving behind their oil fired powerstation and the fragile network of saltwater lenses. We'd be rescuing any inhabitants from the ravages of tropical storms at great expense to the taxpayer within a few years. I dread to think what the cost of shipping everything in and all the rubbish out will come to.