It seems sensible to me to restrict vs mode to occasions when you have a thrust margin capable of maintaining speed. At high altitude in the climb loss of speed is likely to compromise your buffet margins, not a sensible place to be using VS imo, and in descent a high r.o.d. in vs can cause overspeed. Who cares if the aircraft porpoises a bit on the final climb? Does it matter? It probably indicates you are aiming for a level close to the aircraft's limit (esp airbus) anyway, and then surely you need to protect your speed above all.
My general rule of thumb was if there was spare thrust (in either direction) vs was available, once thrust was at or close to either max or idle it wasn't the right mode.
OK, sometimes you'd use vs to bump a sluggish airbus up the last 100ft or so at the expense of a couple of knots, but not much more.