Of course there are a multitude of factors, and things that make it extremely risky for pilots of light (or very light) aircraft is the lack of momentum, and the lack of speed above the stall.
Take for example something like a Jabiru or Skyfox. Climbout at 65 KIAS and have an EFATO. Not only do you have to rapidly lower the nose to the descent attitude (and we all know what that looks like, right folks?) but you will have already lost speed before you got there, and will have to lower the nose even further to gain it back - in this example to a 65kt glide.
Then if you want to do a steep glide turn you have to lower the nose even further due to drag, maintain balance, be aware of the increase in stall speed and look where you are going. Many qualified pilots have difficulty doing these things in a normal powered steep turn.
Now add stress, the 3 second 'Ohfark' delay factor, briefing the pax, checklist items, securing your harness, opening the door etc, and many of us simply don't have the time or capacity.
By all means give it a go if you think it's worth a shot, but be brutally honest and realistic in your self appraisal of your own ability, and the flight envelope of your aircraft.