Keep going and don't give up old chap!!
Get your paperwork together. If your CAA doctor refused the initial issue then this should go to the CAA who should then consider it. If they then refuse, you can ask them to look again and at the same time provide extra evidence.
This should be reviewed by a different person than the one who did the first assessment, NOT the same person.
If this is the refused you can go to appeal, where you are in full entitlement to attend with whoever you wish. The panel should have no one on it who has been involved so far, and some of them will need to be experts in the field of medicine in which you are being refused a medical issue.
There are some great CAA docs out there and I doubt any of them refuse a medical on grounds other than cautious or obvious reasons. Some even offer to help with getting your medical sorted. If you are not happy with the CAA doctor you used, find another who will. There are lots of good guys out there.
In the end...it depends how badly you want it. It sounds like you know where you stand medically.
By the way, the CAA Medical Dept is worked off its feet. They do get things wrong.
