I agree with wearing a lifejacket and I always do. ( I somewhat assumed that as read, assumption is the mother of...)
However I also agree that I would NOT normally inflate it unless I had to.
I had a number of experiences with "automatic" jackets in the marine environment - it is particularly difficult to climb on board a rib/liferaft on your own with a jacket fully inflated (as I am sure you are aware).
The issue with being cold is to do with the thermal properties of the sea. Once out of the sea, whilst you will be cold and wet you will no longer be loosing your body heat trying to warm up the sea! So you still have the significant advantage.
PS. I also carry the survival product liferaft - but am somewhat nervous of its efficacy - having spent many years carrying robust deep ocean certified liferafts!