Climb rate, or rather lack thereof, on one engine is the primary reason.
Unless the runway is very short, the use of flaps for takeoff on most (but not all) general aviation single and multi engine piston powered aeroplanes is not an especially good idea.
Modified or original build STOL types excepted.
The reason is quite simple.
Nearly all airfoil sections, whether swept wing or straight, will exhibit improved rate of climb with no flaps versus flaps extended.
This is especially important for light multi-engine piston powered GA types, which have, at best, limited climb ability with an engine failed.
Turboprops?
Well, the Cessna 441 for example does indeed use flaps for takeoff, but OTOH has rather good performance on one engine, thanks to the SHP available.