i.e. unless you have written permission from the owner, you can't land or take off from there
That is not quite true. I regularly (in the summer anyway) fly from a closed airfield, out of hours. We have a verbal agreement that residents can come and go up until sunset or after sunrise despite the AD being closed.
Insurance is not an issue.