Where I fly in class D there are specified entry and exit lanes so that we dont get these problems. Also, I was taught ( at a different airport) never to fly directly over a VRP, always to keep it to the left. This would stop (hopefully!!) potential conflicts between inbound and outbound traffic.
Rule 5 states (for a congested area) that you must not fly less than 1500ft above the highest object within 600m of the aircaft. If the area is uncongested then the 500ft rule applies.
I reckon you should stay as high as allowed for as long as is safely possible. No point in descending to circuit height well away from the airfield, especially if it is over congested areas because you may not be able to glide clear.