Many Skippers have advised me that it is okay as long as we understand what we are doing and conscious of the risks and really mind the shop especially at higher altitudes.
However from experience, looking from the risk management perspective, as long as everything is normal when both pilots are aware and consistently paying attention to this matter it's really no big deal. What about times where you are presented with problems or ECAMs etc, it is very easy to let that slip off your mind when both of you have your mind on the issue at hand. I know it says Aviate first, but I've been caught myself with us busy with the ECAM and asking for weather deviation etc, and the speed meanwhile is dropping as the aircraft is climbing away.