Understandably this will worry you but don't let it if you can. My AME said he is not aware of anyone who has had their medical pulled for poor hearing alone. If you start to really struggle on audiograms with the AME then the next step is that you would do a trip with a line trainer who would hopefully certify you as still being competent with listening/comms and you will be allowed to continue to fly! If you had the volume on the audio panel turned up fully and you were still struggling to hear the R/T then I would be worried - otherwise put it behind you. I presume that the uncovered ear is the one that came off worse?
I think the CAA and the AMEs often forget that we work in environments that can be noisy and some employers might not take enough steps to look after our hearing etc etc. How many turboprop airlines provide noise cancelling headsets for their crew to use??
I think the current rules are a bit old in the tooth - all we need to do is to be able to hear normal conversations in both ears. Afterall when do ATC whisper a instruction to us???!!!