Send Clowns,
My last post on this as it's getting a little repetitive...
There should be no conflict, because although there is no formal requirement to be in contact with ATC, there is a requirement to 'conform with the pattern of traffic'. If aircraft conform, they will not conflict!
Yes, in theory, you could fly just outside the ATZ without talking to ADC, but I don't believe you'd be displaying much airmanship.