There is a very upscale restaurant in New York City called the Picholine which takes just American Express cards. OK, they can afford to, but there are a few merchants around taking just Amex, and they are obviously exceptional.
As I understand it, the usual commission in the US and UK for Amex is around 5-6%, against half this for Visa and Mastercard.