That is actually a common misconception and it can be deadly. Swimmers think that by swimming to shore they can keep themselves warm. Kayakers often take off gear when they are hypothermic, because they are working hard. I'm not a doctor so I can't explain in scientific terms but my understanding of it is that apparent heat that you feel is being generated by your internal fuel supply which is limited and the same fuel supply that powers your brain. The sweating that you do in such a situation makes it worse (one reason that we call cotton "death cloth".) Many people in the later stages of hypothermia have been found in the snow with their clothes off.
Stay warm!