FAR 61.75 has some info on this..
However, it's FAR 61.19 that addresses your question. It says:
(c) Other pilot certificates. A pilot certificate (other than a student pilot certificate) issued under this part is issued without a specific expiration date. The holder of a pilot certificate issued on the basis of a foreign pilot license may exercise the privileges of that certificate only while that person's foreign pilot license is effective.
So whatever you heard (even from FAA inspectors) contrary to this is not correct.