You ought to get better treatment than others, for example access to better seats on the aircraft (on BA flights some seats are blocked for BA Golds). You ought also to be quite high up the list for any operational upgrades on BA flights, too.
If you don't like the lounges, then it's fair to say that you aren't getting one of the key benefits. But - irrespective of the quality of your fellow pax - are there really any better places at the airport to wait for a couple of hours, with a glass of champagne in hand? For me, probably the best thing about the lounges is being able have a shower after a very full day at work, to get comfortable before a 13-hour+ overnight sector in economy.
And the miles: Agreed, not much use if you don't want to fly on holiday. But if you are flying on holiday anyway, upgrade awards are a good way to spend the miles.