Wouldn't the answer vary between aeroplane types? On mine (80 HP high wing 3-axis ultralight) I would certainly avoid flaps until I felt certain the extra drag were acceptable. The one disadvantage being that abruptly going from zero flaps to full flaps might require some quick and hefty elevator action.
And by the way, I don't see why such an experience should be "unsettling". To me it would simply be an incentive to do it all over, until I get it right. Not that my PFL exercises are always very impressive...