Well, indeed.
I used to fly a small drone. I was never closer than several miles from the nearest airport and I kept well within CAA rules (below 400', never more than 400 yards away).
Yet on a couple of occasions a helicopter came by at about 200'. In each case I descended rapidly but I can't be sure that if the helicopter had been going straight at my drone I would have been out of the way in time.
Eventually I sold the drone. I got some great aerial footage but it wasn't worth the risk of a collision.