Briefly.........Vne (and what the airframe "feels") is indeed determined by TAS and not IAS. Vne actually decreases with altitude and if you hot rodded your engine and took your plane up high, you could actually encounte Vne (and flutter) below the Vne marked on the ASI. I guess it is marked on the ASI for a sealevel IAS, because they know that up high the aeroplane as designed would never (in normal flight) reach a TAS which would exceed Vne (due to power limitations of the engine etc). All bets are off if you decided to tinker with your engine and add a Turbocharger (for example) without any further modifications to the airframe.
It is to do with the speed the molecules of air pass over the control surfaces and not the density of the air which will determine when flutter will occur (the reason for Vne).
Here, this explains it better than I can
http://www.vansaircraft.com/pdf/hp_limts.pdf