A lot of people fail the Isihara test, but that test doesn't test colour vision in any meaningful manner. Its only plus is that the equipment is dirt cheap.
There are a few other tests, of which the Wright-Holmes Lantern test is regarded as the only one which means anything much (according to the CAA-sponsored study which I probably have somewhere). You can easily fail the Isihara almost totally but get 100% in the W-H LT.
And you need to pass the CV test only once and that is good till you die, worldwide. (Obviously it isn't good after you die).
No CV means no night flight but you can still get the IMC Rating, and I have heard you can get an IR too (with a no-commercial restriction).
Go for it.