If you decide to go into Oslo itself, the local trains take five to ten minutes longer than the Flytoget (the Airport Express), as they stop in Lillestrom, but are roughly half the cost. There are plenty of interesting places walkable from the main station and the trains are regular and reliable, so you could easily have two hours in Oslo without too much risk.
If you stay in the airport I'd agree that, largely, the Duty Free isn't much cop (different story if you're staying in Norway) but everything else is so expensive that it can make sense to buy stuff like chocolate and drinks there and then consume them!!
Oh, Gardemoen is utterly crap for vegetarians (although not as bad as Stavanger Sola). This is not true of Norway generally (well, not Oslo or Rogaland, with which I'm familiar, anyway).