Because in order to have the right conditions for safe manouevre you need sufficient RAS (to avoid stall, to have a safe margin in turns, etc). Therefore your RAS is pretty well fixed for a given manoeuvre. However, at altitude, TAS is higher for a given RAS. As TAS gets higher, rate of turn for a given bank angle becomes less and radius of turn for a given bank angle gets greater. Result - less manoeuverability.
Simple, really.