jayemm, I wouldn't be too hard on yourself, as it sounds as though the other aircraft may not have been paying attention, either by looking or listening. He may have made his somewhat petulant call whilst still miles behind you. Some people seem to fly as if in a bubble and only listen to their own RT. I mentioned above that I'm not generally very keen on orbitting in the circuit, but recall once doing so in a slow-chugging Cub, with an instructor. It was his call, as there was a faster aeroplane behind us which appeared to be having difficulty slowing down and keeping a distance. The instructor clearly announced that we would orbit once to allow the other guy to pass ahead (he was simply being courteous) . The next we hear is a splutttering alarmed voice on the radio: "There's somebody flying the wrong way around the circuit!". At least he was (sort of) looking, if not listening.
Saab, if going around from downwind or base, I would call "going around, circuit height" and fly the normal base leg without descending, then fly along slightly to the right of the runway and turn crosswind. Lookout and listening should tell you if anyone is joining deadside or coming onto crosswind that you need to avoid. As keef mentioned, if there is ATC, you may get some help: they might for example invite you to peel off for a a tight circuit from half way along the runway.