I always leave about 2000' between myself and the mountains, unless flying in a pass, then I try and leave 2000' AGL to the floor of the pass.
Sometimes it's better to be low. Usually the wind is stronger at altitude, and you can sometimes encounter severe turbulence which you would have avoided lower down.
When I was learning mountain flying in the Alps, I once flew on a very windy day with an instructor. He asked me to cross an 8300ft pass and told me not to exceed 8500ft. I thought he was crazy, so surreptitiously climbed a bit higher to give myself a safety margin. As we entered the pass at 8800ft, the turbulence was so bad that I almost lost control. We did a 180, descended to 8400ft and tried again - and got through with no problems.
As On Track says, the answer is to do a mountain flying course. A lot of conventional wisdom doesn't apply in the mountains, and sometimes what seems safe isn't.