I don't think that a UAV is likely to be any more of a danger to me than a Cesna 172 being flown by the new breed of magenta line cowboys, both have next to no time for "lookout".
At least the UAV is going to continue on track if you see it at the last moment rather than turning unpredictably.