I agree that it is not completly safe but a go around at the wrong time is not safer. Landing without clearance should always be a very rare special case, not a procedure.
What most people do wrong imho that when they receive a very very late clearance, they try to read back or even recapitulate the landing check list or whatever procedure you have (starting a challenge/response wording). At this moment you fully concentrate on the landing, no word spoken (unless relevant to the landing or the go around).