On the classic you are right, there is sometimes a thrust lever split. However there shouldn't be with competent maintenance, so if you observe itwrite it up and let them correct it. Once you switch to the NG, thrust lever split is a thing of the past during normal flying.
Of course you can fly without a moving thrust levers, many airbus pilots do that every day. It just is bad design as we only have so many input channels and the visual tends to close down pretty rapidly during non normal situations (tunnel vision) where the tactile channel might still have capacity. Airbus took that one needlessly out of the equation putting even more stress on the already overstressed visual input channel. However that is only one part of a pretty big puzzle.