As others have said already, an aircraft will remain in the air and perfectly controllable with all sorts of bits open / hanging off. Distraction however is certainly be one of the worst contributing factors to an accident if you let it get the better of you and there are numerous examples of both private and proffesional pilots dying because of it.
I've had doors come open or partially unlatched on Pipers and Cessnas, a generator failure on a night flight, a carb heat knob and cable come off in my hand, a laser pen attack at night, a radio failure on short finals and a variety of other weird and wonderful equally minor incidents. None of these were of any real drama at the time but any of them could have become fatal if allowed to distract from the task of flying.
Having said that I did allow myself to get distracted whilst solo in a JP3 several years ago when the U/C failed to retract after take-off despite much mashing of the U/C retract button. Whilst fiddling with this the sleeve of my flying suit got caught on the throttle bringing the power back without me noticing. The JP3 is lacking in power at the best of times but is a real dog with the gear and flaps still dangling and I was fortunate enought to check the ASI just as it was decelerating through 100kts..... Full throttle and lowering the nose quickly rectified the problem however a few more seconds and it might have been a very different matter had the engine spooled down much further and the sink rate kept building...
There but for the grace of god and all that.......