Ok, after hearing aircraft flying over my house the whole afternoon and paying close attention to them for a change I am now convinced that it is the speedbrake. I even had a chap who deployed, retracted and deployed them again (as I have seen some colleagues do when selecting CONF1 with the speedbrakes out), with all the howling that comes with it.
Also in the flightdeck you can hear this howling at certain speeds when moving the speedbrakes.