Words, phrases and terms rely on their utility by being understood and interpreted by those who use them and those who interpret them.
Those who object to the use of brevet when describing a uniform badge denoting qualification understand what it is being used to describe (a qualification badge). So their objection is not through a lack of understanding of meaning in context. If the meaning is known where does the problem lie?