Yep, by all means don't accept visual approaches at all. There's nothing wrong with that. But DO know WHY the controller is ticked off at you. Know why you're getting the treatment. Most don't know why they're being treated that way. That's my point here, nothing else.
Bye