The "annoyance factor" is hard to quantify but, to the first order, a decent sound level meter with a dBA weighting (which accounts for the response of the average ear) will help. Unless you get a really bad one, most modern meters based on small capacitor microphones are actually pretty accurate in my experience, at least below 10 kHz. If you can take a few readings around the pilot's head position, then take the average, it'll give you a decent impression. (Strictly, your head modifies the sound field, and you shouldn't average dB..but don't worry too much about it). If you're doing it yourself, or have a really obliging friend, take some dBc weighted readings as well: that gives an idea of the power with a more uniform instrument frequency response.
Typically, there is a big difference between old and new aircraft. Of the spam varieties I can recall, the old Cherokees were shockers. Well designed aircraft using Rotax 912s are actually pretty good, with and without ANR. From personal experience, if I compare a 1966 C172 with an 0-300 with a Tecnam/Rotax P2008, the Tecnam wins hands down.
ANR can produce some odd consequences, too. Just like on commercial flights when your noise-reducing headsets reduce the jet engine noise so that the inane conversations of other passengers more effectively enter your brain, you can get similar results in your own aircraft. My Sennheiser ANR is so effective that I now spend my time looking for hitherto inaudible air leaks around the canopy.