Christchurch!
Mate, the industry in NZ is probably smaller than you're expecting, but nearly all of the stuff you'll enjoy seeing out your perspex window is on the South Island, and Christchurch is in the middle of it. There's a successful Aero Club based at the (Intl) airport. My strongest advice: if you go to Milford Sound, take an experienced local instructor/charter pilot (from Queenstown or Wanaka perhaps).
There's a reasonably priced PA28 at Wanaka too.
Enjoy.