I'd say if the seatbelt sign is on, then fasten them as soon as you are settled but if it is off, then definitely don't fasten them as this will usually mean the aircraft is being refuelled at the time. We make regular PA's to that effect if we do board during this time for exactly the reasons you have stated. I'd rather passengers did have their belts on before push back as this can sometimes be quite jerky. It's been enough to land me in some poor chaps lap before now!
Hope that helps.
JSL