Long before getting to the base to final turn, make sure the trim is set to the right setting so that the elevator is at neutral load before going in to the turns. Is it flying hands off downwind? If it is, OK, if not, sort it. If the trim is wrong, then that's one more thing that can cause problems, as the loading can end up feeling all wrong. If it's right, then if you let go as such, it's going to come back to something close to where it should be, which may be the difference between comfort and strife.
don't mess with flaps in the turn, get them sorted out wings level, and trim again. That way, you've reduced your work load.
Flying close to the ground is inherently more dangerous than flying at comfortable heights, so the concentration levels have to be higher. If you've passengers on board, shortly before entering the circiuit, politely suggest something like "it's about to get busy now, please don't talk to me unless I say something to you, I need to make sure I don't miss anything from ATC. Thanks." You might need to vary that a little if everyone is on headsets, but I'm sure you get the picture. Then, don't talk to the passengers unless it's something to do with their safety or similar, and concentrate on making sure that it all happens the way it's supposed to.