Think of Restricted Areas being Class C airspace. When they're active a clearance is required to enter/transit, when not active it's simply Class G airspace. As Arm Out said, if you weren't permitted to use the airspace when the Restricted Area was not active, there'd be no point in having hours of activation. Obviously, if you know the area is to be activated when you were planning a flight, you would make sure you were clear when it was due to become active.