You were put in the unfortunate position of trying to read the Controller's mind.
Even when I have gotten extremely busy, far busier than what you described, I would make sure to at least let you know where I wanted you next. Either an orbit or two over the vrp or something, but to let you hang there is just not right, with all due respect to fellow ATCOs.