Perhaps. It is more difficult than other routes to professional aviation. Predominantly because it asks more of you as an aviator and a person. You'll likely end up in charge of not just an aircraft but people and serious situations. I think from having seen quite a few students pass through ADF pilot training, that those that aren't fully committed tend to be more likely (not assured) to falter. It is certainly a route that requires total commitment and a fair amount of drive to be successful but it is also within the capabilities of normal people. I wouldn't go down the Military road if I wasn't sure it was the right way ahead. My observation was that the most successful students were those with the most drive and not necessarily the best skills. Hard workers tended to receive more time and effort from instructors (above the norm anyway).
Another note on a previous comment - if you go through ADFA you will gain a Return-of-Service obligation and if you fail pilot's course you may be required to serve out this obligation in another role.
Mate, if you want to work hard and do the best flying available in the country, don't doubt yourself - have a crack!