Do I just forget about trying to be safer pilot and stay a VFR duffer with an extreme phobia of stumbling into weather
Given what I've just read - and if you've tried a few flight schools and you're really struggling on IFR theory and practice, then I guess the answer to the above is yeah. Sorry. I think some people have to come to grips that they just ain't cut out to do it!
On extreme phobias you can manage them better by flying more with other more experienced people, VFR and IFR, so that you can understand the weather better in their decision making 'fly or no-fly' process as well as understanding the objective risk of entering IMC better. If you feel your phobias are unmanageable and you want to help it you can also try taking up psychotherapy.