TOGA won't de-stabilize your situation, if things are going well, though until things are under control, leave it at Flex..
..as for the advisability of taking TOGA - do you really want to potentially miss an obstacle by 35 feet?
When in doubt, use the thrust...