I agree, but oddly the CAA (and I believe EASA) don't - as they only require a competent observer if you practice IFR without a view limiting device, but an actual second pilot if you do use the view limiting device.
Makes sense to me - removing foggles, for example, involves being very careful not to pull my glasses off at the same time, and usually involves removing and replacing the headset. I'd not feel happy doing this in a hurry (if there's no hurry then it wouldn't be necessary to do it at all!) low on the approach on a collision course with something else with a non-pilot sitting next to me.