I don't see how you can justify following the missed approach of the ILS for that runway if the ILS is U/S
Agreed... wouldn't you then fly the missed approach procedure for the instrument approach in operation (VOR or whatever)?
In the circumstance raised above where you've been vectored for an ILS approach, chosen to finish visually and then the ILS goes off the air during the visual segment; if part of the missed approach procedure is dependent on the ILS DME, for example, I would probably ask for vectors from ATC. If the missed approach didn't depend on any element of the ILS, but instead on other navigation aids, we would probably just fly it as briefed.
At my home base we do visual approaches regularly, and at no point do we ever brief a different missed approach for a visual completion than for following the ILS all the way down. The missed approach altitude is set the same, and the procedure in the FMGC is unchanged from the missed approach for the in-use instrument procedure.