I suspect that if you looked closely at the stats for, say, the Lightning or Harrier then the 104 would not look so bad.
I would guess that the losses were the result of several factors. The GAF had little experience of sophisticated aircraft after reforming post WW2 and the learning curve would have been very steep. The aircraft was pressed into service as a low level bomber when it was designed as an interceptor and it had just the one engine, so not much redundancy in the event of a bird going down the intake. The wing area was small so landing speeds were high should the flaps not go down.
I had a trip in a Dutch 2 seater and I thought it was a rocket ship. Great fun and excellent accelaration
And the noise the J79 made was wonderful!