My understanding would be :-
Class A, B or C airspace - Pilots may adjust their speed as they require without reference to ATC unless flying a procedure which specifies an IAS (e.g. Sid / Star) or, a speed restriction is placed upon them.
Class D Airspace - Below FL100 pilots must fly at 250kts or less unless authorised by ATC, and adjustment below this speed is acceptable unless flying a procedure / ATC speed restriction has been placed.
Class E F G airpsace - Below FL100 pilots must fly 250kts or less, ATC are not allowed to increase this speed restriction as the see and be seen principle applies. Pilots may adjust their speed inside this bracket as required. If ATC / procedure specifies a speed limit, this must not be deviated from unless permission is sought from ATC.