The Vanguard's only appeal was to extremely academic bean counters who only looked at seat-mile costs. And there were even less of them around at the time than nowadays. A 140-seat short haul aircraft was just too large in 1960, and all 140 of those seats just could not be consistently sold at sensible prices all day long. Of course, those were the days of fixed and regulated fares, so no cheaper offerings (apart from night flights) to encourage demand, so they did a lot of either flying half empty or sitting around between peaks. It was another 10 years before BEA got jets of even the same capacity.
Quite a bit of an aircraft operating cost is crew, of course, and if they are paid by the flying hour (as someone on an x hr/month contract effectively is) then an aircraft which is notably slower will end up with greater crew costs, to an extent that offsets other operating costs.
It was also a thoroughly uncomfortable aircraft, not only seated high density but well known for vibration, inside and out, and was a very bad noise neighbour. You could hear them taxying miles away (someone wrote that taxying at Ringway was audible in the centre of Stockport), and having lived at the time at the midpoint on their London-Belfast run, they were plainly audible from the ground. I believe the key noise issue was with the propellers. I don't know how De Havilland (who made the props) could have got it so wrong while Dowty (who made the Britannia props, which must have needed to absorb around the same horsepower) managed to get it right at the same time.
Plus it was propellers, not jets. When all the European competitors started moving to jets, starting with the Caravelle (introduced before the Vanguard), it was just commercially uncompetitive. Sure the Vanguard had a lesser seat mile cost, but overall aircraft cost for the smaller Caravelle would be around the same. And it was faster, and A JET. This was the jet age.