From a poor memory:
If I remember rightly it is because it works of binary data.
There was only space for
12 characters. Each squawk number was made of binary (1's or 0's). As there are only characters able to be transmitted, that meant you only had a max of three digits for each squawk number.
With me so far??
As the max number of representations in binary using three digits is 8, (000, 001, 010, 011, 100, 101, 110, 111), the number of usable numbers in a squawk are 0, 1, 2, 3, 4, 5, 6, 7
So, a squawk of 4125 would be represented as
100,001,010,101
(in fact there may be 15 total characters including the spacers (the commas in the binary example)
If 0-9 were used in binary, then 4 binary digits would be needed, which would mean only three characters, leading to a max number of squwks of 999, as opposed to the 4096 that are currently available.
This is all from memory when at the College of Air Traffic Control. So it is may be wrong. If it is right, thanks to Pedro!!!