BNEA320 - most of the ones that could get over 400ft (taking into account ability to handle wind, endurance, and visibility for the operator) would weigh 300g or more. 300g probably isn't a big deal for an airliner, but for a GA/RA plane it might be worrying.
You can go up to around 3kg before they start getting relatively expensive. 3kg is definitely a major concern for GA/RA; I'm not sure how much it would bother a larger plane. Aluminium or carbon-fibre bars going through the engines can't be good, in any case.
The really tiny ones (sub-50g) aren't much of a concern. Too light to do much damage, and almost impossible to fly anywhere other than indoors. The really big ones probably aren't a major concern either, because they require quite a lot of effort and money to construct and run. This tends to result in them being flown by more mature pilots.
Snakecharma - the regulations also say that they must be flown so that, if one component fails, the vehicle will be able to avoid a populated area. Since a quadrotor with a failed motor falls more-or-less vertically (they're completely uncontrollable) this implies that they cannot be flown over populated areas at all. An octocopter (which can survive a failed motor) should be legal.
All the model aircraft pilots I've met just consider 400ft to be a hard limit, regardless of location. Much easier to do that than to try to understand the regulations, and realistically 400ft is high enough that it's getting hard to visually orient the plane. The only time I've seen someone go higher was with a high-performance glider, and for that they got explicit permission from the local airport.