If you're at the point of wondering whether your track miles are 'too few' even considering full use of speed brake then you may need to give more consideration as to whether you are likely to be able to achieve a stable approach at all.
Remember that, effectively, RoD is a 'True' Air speed, in ft/min. RoDs are higher for the same config at altitude (i.e. when the angle of descent is the same) since the aircraft has a higher TAS.
Personally, as well as the 3 x altitude rule of thumb as above for normal circumstances, anything less than that will require speedbrake, any more than about 2.5 x altitude will need prompt full speedbrake and anything near 2 x altitude needs full speedbrake, maximum flaps and gear extension (very high chance of a rushed/unstable approach and definitely better to ask for /fly extra track miles).
Of course, it all depends where you are in relation to the field - 40nm out at 20000 may give you time to make use of an increased rate of descent, 3000 at 6nm: forget it!