Pace,
I am 99.9% sure that the single pilot RVR limit is an absolute limit at the time you are executing the approach. So if the AP packs up enroute and at the FAF the tower says RVR 700, you are in the same position as you would be if the AP is working and the tower says RVR 450 (I.e. 100 metres less than legal minimum and illegal to continue to approach in both cases). So yes, without doubt, I personally would not continue.
You are of course correct when I am practicing I do it to system minimums (and sometimes consciously below just in case one day I have a really serious emergency shortage of options). However, I am a chicken and have a choice of where I fly so in a minimally equipped IFR aircraft with no backup systems, I would be a lot more cautious of the mission profile than in a well equipped aircraft with plenty of redundancy.