Learning to fly isnt too dissimilar to learning to drive so it isnt fundamentally difficult. However, there is a difference between your average driver and a formula 1 driver, so I think there are definitely people out there who are more 'natural' than others.
I dont have any experience with the civvy world where, in comparison to military flying, there is less emphasis on basic stick and rudder skills as opposed to being an efficient operator.
I have seen a big difference in the standard of pilots throughout my RAF training and there are definitely those who are more gifted than others, and quite often this is borne out of confidence.