Why are you assuming it's some kind of radar system?
With current computing capabilities, I'm sure that some form of visual recognition system using electro-optics and scanning against a database of images of planes of interest could be cooked up. Image recognition is already in use in the law enforcement area, I'd be amazed if it hasn't been tried for aerospace purposes, there are enough SAMs with visual tracking systems to give it the impetus.
Similarly, one could envision using IR sensing to build an IR image of a plane and do the same kind of analysis in the Ir-spectrum.
In the radio frequencies, you could either use some kind of active radar system and try to identify the blip from characteristics - the ability to count fan blades or whatever with sophisticated process of the radar response isn't exactly a big secret.
Or you could just rely on the aircraft's transponder TELLING you who they are, of course!
(And I defy anyone who doesn't already know what they are doing to build anything from those descriptions; hell, they can go read any Tom Clancy-esque novel and receive a far more detailed description of almost any 'sexy' military system out there ...)