No, you are not right to worry.
The aircraft departed with a certain amount of fuel. The crew made best use of that fuel. They got an approach at the destination and then a diversion to the alternate.
They could have carried more fuel in which case they might have had an approach at the destination, held for a while, done another approach, still not got in and diverted to the alternate.
Neither of the above scenarios is any "safer" than the other. The second scenario has the benefit of increasing the chances of you landing at your planned destination but the downside is that it costs fuel to carry fuel and carrying extra fuel can also mean not carrying as many passengers.
Ultimately if there is enough fuel to ensure that you can land somewhere then safety is covered, anything else is just about passenger convenience. If you want to worry that you might end up at an alternate airport, then go ahead and worry, but there is no need to worry about running out of fuel.