The symbol, in all iterations has been around for many thousands of years used in many different world religions. The National Socialist Party that came to power in Germany decided to tilt it and make it their own.representing the qualities desired to defend and put forward the cause of the Aryan race. Sadly due to the horrors of WW2 and the emotional charge instilled in people because of this association ANY Swastika is immediately associated with the Nazi Hakenkreutz, no matter how different in design, colour or form.
About 30yrs ago I had the experience of watching a friend give a slideshow presentation to a room of 60-70yr old Germans of his mountaineering expedition in Nepal. During the slideshow there were a few images of Swastikas on buildings, monasteries mostly, and there was an audible collective gasp by the audience and a following mini-hysteria that lasted several minutes. This was caused by the emotional charge associated with seeing this symbol again. Once pointed out that it was over 1,000 years old and was a sign of harmony and auspiciousness and was different in every way to the Nazi version, the room settled down again.
Whilst I agree that as a symbol, the associations are too strong to be workable because of sensitivities, it is a shame that something that has been around since before the Nazi Party, in the case of the Finnish and Latvian forces, and for millennia in religions should be quashed.