The colour does matter! Non NVG compatible colours/reflective tape etc will cause reflections in the windshield that can be seen through the goggles. The importance depends on how you operate. The worst case being if you are operating under very difficult conditions, e.g. over the sea with a total cloud cover (IMC, but with some visibility). Under such conditons you will be able to see reflections from white paper on your kneepad, of your hand (if you remove your gloves), etc.
However, for operations under most VFR night conditions it will not be a major factor.