Been there, done that.
We had an AoA and Beta vane on ZF510, the BDN / ETPS Tucano. In initial testing, the damned thing oscillated constantly and was useless.
We got the resononant frequencies of the vane by putting the AoA trace from the FTI through an FFT analyser (which strangely enough gave the same result as my putting a ruler on the trace printout) and confirmed it by doing a bonk test.
Assuming that the vane boom was a simple mass-spring-damper system, we calculated a change in length (about a 200mm reduction) which adequately changed the resonant characteristics and the device was then perfectly useable with virtually no vibration.
If you are starting from scratch, you want to know the airframe natural frequencies, which the manufacturer may be able to give you, but failing that you can do a bonk test on the airframe with the engine running or ( much better) fly the aircraft with a recorded on some appropriate motion sensor (if you have a gyro platform that'll do the job). An FFT (Fast Fourier Transform) analyser will give you all the frequencies, although it is possible to get the main ones from the geometry of the trace. Get the drawing office to predict the resonant frequency of the boom with the AoA sensor on, and ensure that the period is at least 30% different to any of the airframe modes.
G