From a
bureaucratic point of view, the
regulatory special use airspaces (Restricted and Prohibited) are the only two designated with defined rules/regs
specifically by wording in Part 73 (Subparts B and C)
https://www.law.cornell.edu/cfr/text/14/part-73
From a
practical point of view, the
regulatory SUAs are the only ones that
must not be entered without prior authorization or advance permission from the Controlling Agency (FAA) or Using Agency (other Government agency - armed forces, security agencies, NASA, etc.).
Restricted SUAs have slightly looser rules: permission to enter
can be granted by the FAA, in addition to the non-FAA user agency; and may only be in effect during certain published times.
Note that Authorization or Permission are
not the same as getting, for example, a Clearance into Class B airspace, which
may be given on the spur of the moment, if one pops up on radar.
One needs to have made arrangements to fly in Restricted/Prohibited airspace well ahead of time. Examples of Prohibited/Restricted airspace include the airspace over central Washington D.C. and especially the President's/Vice-President's Residences. Or the President's location at other times. Neither an IFR flight plan nor "weather avoidance" in any way permits entry into those airspaces.
The non-regulatory SUAs may, in general, be entered without prior permission or authorization. They are simply areas with special
hazards to be aware of (military aircraft operating at high speeds; in-flight missiles or bombs (CFAs), extra-high traffic (fly-ins or heavy-use training areas or glider activity), and so on.