The only downside (aside from cost of housing) I can think of, is that its Island life.
Island life affects some more than others and it's worth considering carefully before you make the leap.
Should you need or want to get away, individually or with your family, it's as expensive as hell on the ferry during school holidays, none of the airlines carry cars anymore.
I would venture that the younger you and your family (if any) are, the easier the transition will be.