Yet you still have to use QNH in addition to the QFE don't you?
Of course there is the local QNH, Region QNH, along with the Standard Setting of 1013 for Flight Levels.
We have found it easy enough to do the mental gymnastics you describe and at the same time be able to use a common reference for Terrain and Obstacles which are all based upon MSL which is a nice fit when using QNH Altimeter settings. All of our charts show AGL Heights for Obstacles thus doing the math for you.
Our rules have us using the most up to date QNH Settings within a 100 NM's at a Minimum while Enroute although good practice is to use one from close by and when operating to an airfield or location close by, the setting for that particular Airfield.
It is all what One is used too in a lot of ways.
I have done both the UK and American method and very much find the UK procedures far too complicated and subject to finding the wrong setting showing up.
If you recall that wonderful Memo about Altimeter Settings and the manipulations required for an IF Approach with a Go Around with the attendant shift of controls then you can agree it can get tedious at times using UK Rules. (The Memo was a spoof on BA's system if you recall.)
If FJ's have problem doing Maths perhaps bringing back NAV's might be a good idea you think?
An example of why a QFE Setting might have some flaws:
