If you are in the UK, please report them all, otherwise the CAA and DfT will lack the evidence to tackle the problem. You should certainly be reporting anything near a traffic pattern or above 400 ft.
Of the drone 'encounters' with manned aircraft (mainly CAT), 92% were above 500 ft and 65% above 1500 ft. The highest recorded event so far is FL130, which was a confirmed sighting by 2 pilots.
The collision studies in the UK suggest that you are quite vulnerable in a light aircraft - a 1kg drone is likely to come through the windshield at normal GA cruising speeds.