But by the end of your downwind leg you'll be 1000' above the circuit, so they won't even see you because they won't be looking up there?
This particular case happened at Lelystad airport (EHLE) which has a bit of an unusual airspace configuration. In principle it sits underneath the 1500'+ Schiphol TMA (class A) and even though the circuit is at 700', there is no realistic way to pass overhead without busting class A and leaving a sufficient margin above the circuit traffic.
It does have an SRZ carved out of the class A though, so the ATZ extends to 3500', but I was leaving the ATZ in the direction of Schiphol so had to duck below the 1500' shelf. Plus the SRZ means there's a lot of activity in the direct overhead, such as aerobatics and helicopter PFLs.
Lelystad also has a mandatory VRP well to the south plus a microlight VRP, approach & circuit to the north, a gliding site to the south-east and a few noise sensitive areas around. So if I were to leave the circuit on crosswind, I would have to fly at least about 10 nm around the field to get on course. That's why I essentially followed the circuit (although I did climb to 1000', instead of the 700' circuit height) and left the circuit/ATZ by simply continuing on downwind.
Mind you, it's not something I advocate as standard but sometimes a departure like this is the easiest and most efficient procedure.