Pretty much correct. DVI can carry analog and digital video, plus USB. The connectors can be analog-only, digital-only, no USB, or everything. HDMI carries digital video plus up to 8 channels of audio. If your setup has HDCP (i.e. copy protection), then you need to use HDMI to the display device.
VERY roughly, HDMI in its various incarnations = more video bandwidth, carries audio, talks more deeply with devices it is connected to, and supports copy protection (without which you may not be allowed to display a full-resolution image).
You can spend many a happy hour dicking around with the various connectors and standards, especially if you bought your kit at different times. HDMI versions increment roughly every 2 years.