I think you are correct, especially if the backtrack is required (if there are no available exits beyond your stopping point). Certainly in the UK, and many other parts of the world, once you have been given a clearance to land, the runway's yours until vacating - it can't be used for another movement. It should therefore be in the airport's (and ATCO's) interest for you to vacate as soon as possible and if there's only one point to vacate at, I'd expect you to backtrack when ready.
The only exception I can think of might relate to a single runway access point: the ATCO might want you to hold beyond that point so the next departure could enter and backtrack in front of you, to increase expedition. I'd hope the ATCO would let you know ASAP if that was going to be required, to enable your planning...