Hi Microburst,
The possible cases are many and in some of them what you say is true. But to keep things as simple as possible don' t bother if displayed VFE or VFE next happens to be the right one according to actual configuration. ECAM actions and paper checklist step by step will make things easy. Table is really straightforward.
P.S. I am sure you already have noticed the last note in the QRH just before the table. Max speed (from table) might be higher than VFE (displayed on PFD). In that case you may follow VFE.