From my own experience it's aircraft size dependent. Anything bigger than an A321 or a B757 usually either the refueller has full responsibility or the ground engineer/mechanic.
There are anomalies. AA air crew tend to look after their own refueling on a regular basis even with their bigger aircraft such as the 787.