If it helps, I flew a PA28 today in the only clear slot (Benson vis was <400M in FZFG until 1300...) and the temperature was never above minus 4 in the air. No problem at all. Mind you, we do keep it in a heated hangar which avoids the frost problem...
Basically as stated, watch out for frost and ice on the airframe before you start (MUST be COMPLETELY clear before flight) and visible moisture (IMC) during flight; otherwise you will be fine.
And watch out for low sun in hazy conditions - all I did was (small RAF style) circuits yet I almost lost sight of the runway several times.
Tim