As most passengers using Cardiff will be Welsh, rather than inbound tourists or business people, it is not surprising the bus carries few passengers. If there are two or more people a pre-booked taxi is usually cheaper and a lot more convenient than public transport.
You would have to live within 5 miles of the airport, for that to be true.