ATX spec is 12V +/- 5%. If you can see wobbles on an AVO, the supply's borked. Max ripple @12V is 120mV; if you've a cheeeep PSU (say a Bestec) the 5V line may have little or no regulation, but be relying on the 12V components. There's a "power_OK" signal from the PSU on pin mumblemumble (it was all a long time ago) but, of course, the cheaper PSUs can lie or be unable to work out their own problems. Again, on a cheap PSU, perceived problems on the 12V line could actually indicate problems on the 5V line: current capacity required at 5V is increasing more rapidly than current required at 12V these days, so that side of things tends to be the weakest link.
I see the specs of your card specifies a 350-watt supply. I bet it's the 5V giving you problems.
It makes my tinkering look very tame in comparison
...but comfortingly successful