So why is there no frost or ice on the flaps? Are they not made from the same composite material?
They are not made of the same material. Spoilers are have a honeycomb core with bonded skins. Flaps are built like other structural parts with spars, ribs and riveted skins.