I know there are aircraft limits but i find that very low (just after rotation) altitudes the flight directors on my aircraft are still very lively an as far as i understand it (i'm still very new to airline flying) the a/p will run to the f/d's very quickly and could lead to some very interesting pitch levels ( i had it once in early line training when the a/c went to 24 degrees pitch and wanted to go higher after i put the a/p if i didn't intervene. poor rotation and poor following of flight ditrectors on my part granted but scary none the less)
I personally like hand flying untill we're flaps up and properly in trim. The aircraft gives it to me pretty much in trim when i take out the a/p i feel it's the least i could do to return the favour.