How low did you fly over said building? If it was 150', no good. If it was 1500', no problem.
It's not only a question of rule 5, it's also a question of airmanship. Given that the building was dead side, why did you fly over it during a go around in the first place?