The "big yellow switch", I suspect for the very few occasions it is required, would be overused, and sometimes inappropriately and accidents as a result.
NoD, this is surely a key area that training should focus upon, if and when such a switch is offered to civilian pilots of Airbus aircraft. Without such focus, a willy-nilly loss of built-in protections could possibly lead to more Airbus accidents, not less - particularly if the willing protection throw-offs were not entirely situationally aware or worse, and more basically, unable to hand fly a transport category aircraft without the wondrous, yet molly-coddling digital assistance offered by the Airbus design.
I should add that this shortcoming of recent pilots, the lack of both basic flying skills and instinctual situational awareness is partly attributable to automation across all types and not the Airbus flight envelope protections philosophy per se.