I'm inclined to put non-stop flights at the top of my list. I like flying and it holds no horrors for me so multi stop flights and changing planes isn't an issue as such (I once flew Washington/Orlando/Tampa/Miami on a National 727 and then connected to a Pan Am 727 Miami/San Juan/Antigua/Barbados/Port of Spain). But as I've got older I've come to realise that take off and landing are the riskiest parts of the flight so why do more than you have to.
Then there are the airlines I will avoid. I think I'm inclined to agree with the EU list plus a few others. BUT if I am in the home country of one of those airlines and my choice is between a road journey and a flight or a ferry and a flight I'm inclined to fly. Trains are a different decision - I shall be watching Chris Tarrant in his new C5 show about railways with a big grin on my face.
Having got that lot out of the way, if I'm left with a choice, I will then eliminate the US based carriers. They're OK but nothing to write home about and some of their security rules worry me.
I'm off to Chile soonish and that's BA to Sao Paulo connecting to LAN on to Santiago and a round trip to Easter Island. A cruise gets us to Buenos Aires and then BA back to London. The alternative was Iberia for the long haul and,quite apart from the extra stop that meant on the way home, this is one time when quality came into the mix. I've no doubt some people here think IB better than BA but I prefer BA.
In the end I reckon it's all very personal and I don't expect many people to make their decisions the same way I do.