It's like so many other bloody irritating and stupid clichés :
At the end of the day
At the top of the hour
Basically .....
The worst use I ever saw of 'end of the day' was in a motoring article :
"At the end of the day what everybody wants is a car that starts in the morning."