Thing, if yiu were working a military unit, they were probably using different charts. On military low flying charts, glider sites are usually depicted with a 2nm avoid radius, so I suspect this may be why the controller thought you were infringing.
I used to spend a lot of time on Farnborough LARS (West) explaining to people that it wasn't a good idea to fly over Lasham below 3700ft on a clear day and still they didn't seem to take notice!