I understand your disinclination; however, I still consider that the person purporting to be from the CAA's enforcement team acted in, at the very least, a thoroughly unprofessional manner possibly bordering on the slanderous.
You should always, as I'm sure my old chum Tudor Owen would agree, require that any such caller should make his allegation in writing. Asking you such questions over the phone would be totally inappropriate as you may unwittingly incriminate yourself - can you imagine the police phoning you up to ask who was driving a car which someone had said they thought might have been speeding?
A chum of mine who had a valid display authorisation once received a letter from the CAA stating that they thought he'd been low flying. He wrote back merely stating "Without prejudice, your letter has been received and the contents noted"! Nothing more was ever heard from them....