Next time if any doubt you could ask them "where would you like me to track via to join the circuit, bearing and distance from the VOR please"
They might say "Track to the 080 radial 5 miles then join downwind right circuit runway 32....."
This gives you specific tracking details from which to join the circuit, it might help?
Although if the RAAF can't workout left or right circuit it may not!!