Fortunately we have a good neighbour pair each side.
I am reminded of the ancient wisdom for old blokes: have a garden small enough that your cries for help can be heard next door, but large enough for a little secluded corner on being caught short with very muddy boots on.
Mine is behind the shed, and formalised by a little sign.