The downside is marked volatility and sometimes a lack of liquidity. After the financial crash it took 7 years for prices to get back up to their previous level. Prices are high at the moment and negative equity is a real possibility. Making a lot of money is also a possibility.
In summary, do you feel lucky?