By all means target an aviation career out of a passing interest and a desire to avoid a desk job.
Just be aware that there are a frightening number of people out there with the same intent who have an awesome insight and understanding of whats involved. Which rather puts you on the back foot.
Not too late to get hooked though - go do a PPL.
Cheers,
WWW