I always understood that Hitler did not really have an intention or a developed plan to invade the UK after the fall of France, hence the Lufwaffe strategy in the Battle of Britain had mixed objectives.
My history of the BoB is a little rusty, although having recently finished an excellent book - Spies in the Sky - about the development of what we would now think of as Tac Recce / ISR, it did she an interesting take on this very issue.
The imagery analysts at the time concluded that the threats of invasion were not backed up by the credible preparations required to successfully mount an invasion. The IAs reported that although there were a number of invasion barges either in the channel ports or being prepared to be moved to the ports, there weren't nearly enough for invasion purposes.
On the one hand, that could I suppose be ascribed to the usual difficulties associated with resource allocation, that infighting between the German Service Chiefs meant that they weren't ready for invasion, or as the OP suggests, that Hitler really didn't plan on invading Blighty at all. I suppose that there has been plenty of research into this very topic over the years, but I would suggest that you don't concentrate so much effort on trying to neutralise an opponent's Air Force if you don't then intend to follow up in someway shape or form.