Some a/c, eg. A310, are approved for takeoff with hoarfrost on upper wing surface over fuel tanks and perform just fine. The operative principle is that a small percentage of non-critical wing area is involved. Contamination of leading edge area is far more critical.
Your a/c may or may not be so approved -- check the manual.
Some countries do not allow takeoffs with any ice at all.
Sweeping may not completely remove hoarfrost. If deice fluid is not available (automotive windshield washer and engine coolant can do as they generally contain ethylene glycol) and hoarfrost covers a substantial area, warm water and careful inspection are called for.