I have been looking a bit deeper and as I understand it, it is the arrival time at the final destination that is relevant. Most of the information I have found refers to a single flight and this is quite clear. It's the connecting flight where it is a bit grey.
EUR-Lex - 32004R0261 - EN - EUR-Lex
Article 2 para (h) refers
(h) "final destination" means the destination on the ticket presented at the check-in counter or, in the case of directly connecting flights, the destination of the last flight; alternative connecting flights available shall not be taken into account if the original planned arrival time is respected;