DON'T WORRY; it sounds as if it's the controller who should be worried. He should not have made allegations against you on the R/T. Make sure you write down everything you can remember about the incident and submit a report to the authorities at that ATC Unit. Everything is on tape and if you are right you have nothing to fear. No dangerous "incident" occurred so the matter probably would not be officially investigated but local ATC management should look into the matter and talk to the controller and advise you of the results.