It is considered a facet which does not change with age. The fact that you have an unrestricted medical is testament to the fact that you have at some point passed an approved colour vision test. If they do decide to test you, simply be upfront about it and tell them you have a slight deficiency, but took this test, that test, it cost you a large amount of time and money, but you reached the required standard.
If you've had an unrestricted medical and been flying with it, it's very difficult for them to take it away unless they have a damn good safety reason for doing so. I would say the likelihood of it happening would be slim to say the least.