We currently operate single-pilot ops with (normally 2) police observers. We are day/night VMC only, but with a twin (non-IFR) a/c. Our observers are equipped with goggles, but our cockpit is non-NVG compatible at the moment, so the pilots rely on normal night-flying techniques with the helpful additional input from the fellas who can see.
Part of our remit is to perform casevac at night in life-threatening circumstances, which will normally be in the more remote areas to justify use of a helo rather than a road ambulance.
The transit, weather etc do not tend to give us any serious safety concerns, because our operations manual gives us very clearly defined weather minima (1500' cloud base & 5 km visibility with passengers/casualties on board). If the weather conditions get to these minima, we can easily sack the job & RTB before it gets uncomfortable. The key is making the rules and sticking rigidly to them, no matter what the external pressures are of serious casualties etc etc.
The BIG safety concern is the approach & landing at the scene, because obviously we dont get a good look for wires, FOD, fences, sloping ground etc etc at an ad-hoc site. The key is to take the time to speak to the people on the ground to help select you a suitable HLS, scan it for as long as it takes with a searchlight if you have one, or landing lamps on a dummy approach, and get everyone in the crew to look out for wires (having done a good map recce beforehand).
For us, fortunately, this is a rare occurrence. Potentially, you may be able to reduce the requirements to land at ad-hoc sites by surveying potential suitable landing sites across your area, and holding them in a HLS directory, recording the potential hazards as well as the size, shape, surround etc. , and make this available to your ground colleagues to RV with you at the nearest suitable location.
Hope you find this helpful - Our rules are dictated by the CAA, which makes it easy for us to implement. If there are no laid-down rules in your neck of the woods, perhaps your regulatory authorities should think about it