Tim
The reason you'd probably (well, almost certainly) get away with it, is because no one else is daft enough to do it!
Which does NOT mean it's a good idea!
Also it's not simply a matter of statistics. Pilots don't fly IMC out of controlled airspace, because if they DO find themselves on a collision course with another a/c there will be a sudden noise.
Therefore the lack of collisions is not because it is safe (as I think you allege) but because it is dangerous and therefore only a prize clot would do it!
However, if you are the ONLY one who does it- you should be ok.
If you talk other pilots into doing it, with the help of pprune, you may meet one unexpectedly and not be in a position to post what would have surely been an interesting comment.
Also even if the growth is only tiny- that tiny increase will likely be you!
Which, I suggest, would make a difference to you.