GS pilot - if all you are doing is using your NVG for transit flying at a safe height then NVG may seem straightforward. If you are doing low level, poor weather, unlit and unrecced sites then constant training is required.
Your stats (4 years, 300 hrs NVG) indicate you are flying an average of 75 hours a year which is somewhere in the region of 6 hours a month. That is an acceptable level of currency, especially for cruise flight. You don't say if you are operating single or twin pilot.
Try not doing any for 3 or 4 months and then go straight into a poor weather, low level transit to an unlit and unrecced site.