Yes the F-104 was a failed fighter and found a niche as a low level high speed fighter bomber.
Are you sure you aren't being over generous describing the F-104 as having found a niche as a low level high speed fighter bomber? My recollections are that this was the very role which earned it a variety of less than reassuring nick names;
Widow Maker, Flying Coffin, German Jump Jet, etc and the old joke about how easy it was to make a start in the scrap metal business in West Germany, just rent a field and wait for a formation of F-104s to happen along.
I think it was quickly overshadowed in the high altitude interception role (puttng it in the same genre as the English Electric Lightning, J-35 Draken, F-102, F-106 etc) by later more reliable types such as the afore mentioned F-4, which was a much more versatile fighter.
FB
But again, I bow to the greater wisdom of others here abouts!