Well blow me down. Check out AIP RAC 9.6.2.
That's the one I quoted.:)

The rules here in the US may be different.

If cleared for a visual and then, for any reason you choose to go miss, you MUST remain clear of traffic and clear of clouds. You are NOT permitted to follow the ILS published MA.

This question seems to be debated every few years. In real life, with working radios, the controller will/should issue instructions. I've had a case where he didn't.