Taking away the ability of pilots to cross check ATC's work is less safe than having other pilots cross check the work. There is absolutely no argument to that. Has Montreal Center caused an accident yet? No. Is it possible or perhaps even likely that if there were an accident that the language issue could be to blame? Definitely!
I find it humorous that we have bilingual people coming on here saying it's not an issue, or it's not unsafe. For you it isn't, for others it most certainly can be an issue.
Have you never had an instance where your hearing and understanding another aircrafts clearance has allowed you to do something to increase the safety margin? It happens almost every time I go to work. That same safety net is taken away when flying into YUL or YOW when people are speaking french.