The first paragraph is not really policing you as a pilot, more about me protecting myself in an incident and not issuing what may be construed as an unsafe clearance! See above about "...controllers using their discretion.." In terms of the second paragraph, who's to say that if you are at 1000' over the Lea Valley you are breaking any part of Rule 5 or the alight clear rules. You could consider that you are within the realms of your licence and the law.
The problem with your "protecting" yourself in this way is that it narrows the range of options available to the pilot. As a controller, you may have a picture of the traffic situation, but not of the other hazards that may affect the flight. Only the aircraft commander is in a position to make proper risk management decisions about the conduct of the flight. By refusing a clearance, you may be forcing the pilot into a more hazardous situation: for example, in refusing a clearance across the City CTR you may be bringing the aircraft into more marginal weather conditions, or you may bring it into closer proximity to traffic at Stapleford or Biggin. You certainly increase workload by making navigation more difficult.
Your quote from MATS Pt 1 talks of licence privileges, and is intended to be about visibility requirements, not Rules of the Air. The "discretion" cited in the introduction has its limits: if you personally regarded a particular airline as unsafe, would you refuse its aircraft clearance inbound to City?
I have great sympathy with your discomfort in the current climate of "duty of care", but you have to draw the line between a pilot's responsibility and a controller's. Rule 5 compliance is squarely on the pilot's side of that line.