It's not a situation that I have to handle as a controller at present....and I don't have the books to hand to check....but I have always thought it was the missed approach associated with the instrument approach that you followed (before any visual manoeuvring) that should be flown.
But, as ever, if you're not sure and there are no instructions forthcoming from ATC - ask.