The reason for the 250 knot rule is that, that is the speed decided by the powers to be as the maximum for visual "see and avoid" procedures to work.
In Class A, B & C airspace, IFR Civil aircraft are radar separated from both IFR and VFR aircraft - and thus don't have to rely on "see and avoid" procedures - hence no speed restriction.
In Class D airspace, IFR aircraft are radar separated from other IFR [civil] traffic, but not from VFR traffic, so there is a speed restriction of 250 knots, in order to give a chance for "see and avoid" to work.
ICAO provides that state aircraft (i.e. military ones) are not bound by this restriction. The presumption is that military aircraft are equipped with on board traffic radar that enables them to provide their own traffic separation, those military aircraft that are not so equipped are required by international law to have procedures in place to provide equivalent safety. That means you don't exceed 250 knots in a non-radar trainer outside of a military danger area/ military traffic radar area.