The pilot is responsible for briefing passengers, and issuing instructions. If passengers do not follow the briefing and instructions, they have created their own risk. There can be no moral reason to not provide the equipment proven to save lives - life jackets only save lives if your wear them.
When I took the dunker course, I thought I would be great at it, and always find the lifejacket in the pouch, and take it with me. Fully briefed and rehearsed, I found it once in four tries, and I was not injured, nor startled.
My passenger today wore the lifejacket I provided, and I briefed him to assure his understanding of it's use, and how to exit the plane. I expect no less of myself.