If you don't readback the 'visual' then you haven't accepted responsibility for terrain clearance on the assigned heading.
By acknowledging 'visual' you are agreeing that you can maintain VMC and terrain clearance on that heading, as opposed to a right or left turn where you have some extra manoueverability.
I generally agree that there are too many requirements for readbacks and controllers chasing ones that are not mandated is simply a lack of knowledge or training. Controllers should only require mandated readbacks and those readbacks should only include the minimum essential info.