A practical answer will be that i depends whether you're getting a radar service from AC (or, more correctly, whether you are in a radar controlled environment). If you are not, firstly other aircraft will be separated from you by much greater distances and so a few miles will not normally bring you into proximity to other aircraft and, secondly, the controller will have no way of knowing that you have deviated.
In a high density radar environment where the controller may be using as little as 5 NM separation between aircraft, well, if you're clever enough to have got an IR, do I really need to go on?
Real answer, if you are going to deviate from the track that ATC has assigned or that you've told ATC you will follow - then tell them! You may only have one aircraft to worry about, ATC looks after the big picture.
Slight rant over!