According to various F104 sites it seems only the F104A model and the prototypes had downward firing ejection seats; about 170 airframes from a total build of 2575, as far as I can tell the Stanley C1 seat (downward firing) was modified out and replaced with the Stanley C2 and later a Martin Baker seat.
I believe the high loss rate of F104s was primarily associated with being single engine, highly loaded wing and some pretty nasty spin/stall characteristics.
Looking at the Wiki link for Lightning losses I was surprised at the number of accidents due to landing gear failure. I think the Lightnings problems around engine fires are well known, but it makes you wonder whether a more effective fire suppression system in the engine bay & jet pipe areas would have saved more aircraft and lives.