Its not easy by any stretch. But i'd dare say it's significantly easier if you have a degree or previous training in the field, even if its twenty years ago.
I wouldn't be too sure. If you can sell your aviation degree as a specialised business degree with a bit of engineering (which wouldn't be the biggest fib told in a job interview ever) in 20 years time it's just as relevant as that Bachelor of Business the other guy is doing.
The modern job market is a dynamic place, I know BSc grads working in
PR and
PR grads working in science. And that's only 5 years out of university.
As mentioned above, study because you want to study. University is an awesome experience for many but isn't for everyone. If you can only imagine putting in the effort to get HDs for an aviation course, then study aviation. But don't feel like the piece of paper will set you apart from the guy who did his CPL in 18 months at the local airport.