The old 30-pin connector is a fixed design, with all the pin designations set since the beginning, it therefore has a plethora of pins that are no longer going to be used like Firewire or see declining use like analogue audio out. It would be difficult to reassign them for more modern uses, because of the fallout from damaging some older kit with a 30-pin dock.
Apple clearly wants a connector that allows them to change the pin functions as new things come along. All of the pins on the new connector can be reassigned, with the chip in the plug telling the iDevice what the pins do. So the included USB cable has a chip in it that tells the iDevice which pins have power and which pins are the USB2 connection.
There are already 8-pin adapters for HDMI, VGA, audio. I can see the connector being used for USB3, when the flash memory storage is fast enough to warrant it.