After pressing the buttons, the FD mode doesn't matter.
While I have no doubt about your ability to fly raw data ILS, the correct procedure to use the two buttons on each side is to use them with two more buttons on the FCU i.e. the FDs and switch them off when you don't intend to use them. It's not smart to leave them in open climb mode while descending or vice versa. It can confuse the other guy. Knowing automation well does not indicate you cannot hand fly. In adverse environmental conditions disconnecting the AP at the drop of a hat can and has created problems.