If MH17 was transmitting then surely the missile radar operators knew? A Ukrainian military aircraft would not transmit or spoof on civilian IFF and the missile team wouldn't want to shoot down one of their own aircraft.
If I recall correctly, the rebels did not have any aircraft. As such, they could be confident any aircraft spotted were not theirs. That might (my speculation) have resulted in a lack of care in evaluating the aircraft's IFF status. Or alternatively, some situation may have caused them to believe it was a military aircraft deliberately showing civil IFF codes, or the launcher may have been handed over to a crew with limited training.