BA will not blink an eye lid. You will not have a black mark against you. I would thank them for the opportunity and say your decision is based on the needs of your family and a change in circumstances at your current employer.
Hell they asked me why I wanted to leave my previous employer to which I replied “I don’t I am very happy there”
If they ask why you came to that decision by all means tell them what the factors were. Let’s face it they will understand and probably already know. I know someone who turned the offer down, explained why and apologised for the late notice. BA wished them the best of luck and left the door open in the future.
Good luck with your decision. BA is not for everyone, it’s not as good as it was but it is not so bad that I have left either.