I really feel for you and the predicament you are now in. Avmed are a law unto themselves and have been known to disregard leading SPECIALIST advice regarding pilot health risks. This may take a long time to resolve, and follow you around as you predicted. It may have a detrimental effect on your career for a long time to come.
Here's some advice I gave last time this subject came up ...
NO,NO,NO,NO,YES,NO,NO,NO,NO.
When I do my medical each year the answers never change.
Just answer NO to everything except the one about 20 min exercise.
I have two doctors, a DAME for aviation medicals and a GP for other stuff.
A sad indictment on the Avmed section of CAsA.