ShotOne:
Your assertion is totally correct, Hitler had to go into the Balkans, North Africa and Greece in 1941 to dig Mussolini out of the deep c**p, he'd got himself into. This delayed Barbarossa until the late spring early summer of 1941. Barbarossa failed for many reasons: started too late, should have waited until 1942, too many units relying on horse drawn transport and very long supply chain with few railways for support, but that is a whole other discussion.
Apart from Hitler, Admiral Raeder of the Kriegsmarine was also uncomfortable as he could not be sure of naval superiority in the English Channel and its approaches, and I suspect did not believe Goring, that the Luftwaffe could control the skies, but politically he could not express his doubt. The OKW did not think they had enough invasion barges either.
Remember one part of the British Isles was invaded and occupied, namely the Channel Isles, who were not liberated until after the 8th of May 1945. A dark period of history, of what went on during that time. Read Tom Freeman Keel's book, Alderney to Auschwitz, very illuminating.