My company operates both pax and freighters.
I think a "smooth" pilot (control inputs) shows skill and finesse. I admire smooth handling whether we're empty, full of freight or full of pax and try to be as smooth as possible at all times.
That said, personally, I'm less concerned about in flight turbulence with freight and not just from a comfort standpoint, but with pax you also have working FAs whose safety you need to be concerned about.
JMHO.