The EU may well not allow it, how could you have freedom of movement to some part of the EU but not all of it? It simply wouldn't work. To me, it's either entire free movement (like a Swiss type scenario), or no free movement into the EU. Forget that agendas both the UK and ROI governments are saying they 'hope' for. It's really irrelevant to what comes to pass. you
Denmark, Norway and Sweden have had a common travel area since the early 20th century, very similar in fact to the CTA (UK and Ireland). From 1973 to the start of the EEA in 1994 Denmark was in the EU, Norway and Sweden were not, and it worked fine, so don't anticipate problems where they don't have to be any.