Horses for courses. I have also seen people briefly disengage the AP for some transient maneuver and I have done it myself. Not to mention that on the A320 this is the only way to make use of full speedbrake since it can only extend to 1/2 with the AP engaged. However, it's another matter that you shouldn't do it in cruise since you're in RVSM airspace where regulations require automatic FL keeping.