but it just doesn't seem right to log 4hrs of IF time
Now suppose during at any time on the 4 hour night flight under the conditions you mentioned, with no autopilot and you nodded off for say just two minutes and you suddenly woke up. Chances are in that short time the aircraft (if in only slight turbulence) would slowly drop a wing, climb or descend or worst case go a lot further than that.
On the other hand if you were hand flying on instruments under the same conditions you probably would have prevented any excursions from the flight path. Then in my book you should log instrument flight time.
