First, great to ask this question here to validate your feeling! My first reaction was, if you think of not doing it, don't do.
Could the captain elaborate a bit more on his experience background?
If you had already taken Pax with you and the difference is now wether to take a second it is quite different from first time Pax ops. Further, if this is an airfield you are familiar with and feel your personal margins are low you could choose another field etc etc.