Callum, what type of RAF pilot do you want to be?
Remember that should you be successful in your application, you will be likely be offered what the RAF needs at the time. So if you wanted to be FJ, you could end up rotary for example.
If you don't really mind, go for it, though be prepared to wait some time. Remember too, that you are unlikely to fly a lot and the downside of that is building time and experience - something that will become quite important if you want to continue flying once you leave.
I think it is important these days, to weigh up the 'negatives' as well as the 'positives' involved with joining the RAF, so that you make an informed decision when following your ambition. This includes a Plan B should things not go as planned.
However, as most posters have said, it is worth pursuing your dream. It (quite rightly) won't be easy, but you don't want to be left with regrets.