Sail trim (how the sails are set) on yachts can be a trap - it depends on who's sailing - I've seen some interesting attempts around the Whitsundays on what I can only assume are bareboats.
Current or tidal effects around islands can also create misleading cues. In light conditions the smooth patches can end up on the windward side of the islands.
My tip is to check local indicators (water or other) just before you land. I've seen winds coming from opposite directions less than 10nm apart, and I've been into one port several times in the one day and each time the wind was coming from a different direction.