Instantly crash and burn?
It actually depends on how well-trimmed the airplane is. If trimmed to fly hands-off, it makes little difference in the short term whether autopilot is on or off.
If it's NOT well trimmed, it might start to roll or turn or anything else.
If you read the accident report on the
EAL401 TriStar Everglades accident, the A/P was accidentally disconnected at low altitude, and the resulting deviation was so subtle that no one noticed.