Roxy,
it's bad, and getting worse every year (except for a brief respite during the financial crisis). After a few years of putting up with it our family left with no reservations. Our kids are quite young, and we thought - given the abundant evidence of what these levels and types of pollutants do to you - that it was madness to wait until some symptoms started to manifest themselves before we got our family out.
We acted not based on symptoms you can see now, but on what the future holds for our health. Unless we went into Central or Kowloon we didn't actually suffer too much (eyes, throat, chest), but that is a poor indicator of what this sh!t is doing to you. Remember that not only are the levels very, very high (the authorities refused to use an index used by western countries) it is also the types of pollutants here. I am sure you will do adequate due diligence on your prospective HK employer - for you and your family's sake I would think doing the same level of research on the air quality here would be at least as important.
There are plenty of expats who don't really notice the air - and exercise/hike etc in it to their heart's content. That's fine, but it in no way diminishes the very real risk to your life and health that this air is exposing you to. It may "effect everybody differently" on a day to day basis, but certainly not after years/decades of exposure.
Finally, ask yourself why you would be willing to bring your family an yourself here and accept it. Unless you have previous HK ties, the answer is almost certainly going to be one thing - money. Where do YOUR values lie, no one elses. Are you prepared to expose your family to the clear and indisputable risk to fly a shiny jet?