You were able to travel because there were international agreements in place. Many of these have been superseded since we joined the EU. They need to be re-negotiated - as of now, they haven't been.
The world is unlikely to end, but would you bet your business on it?