If you worry about it it will take longer. Try to relax and don't over-think the actual flying. You don't say why you are learning; for the challenge and pleasure?
Remember the key to being a safe pilot is using your brains not taking to flying like a duck to water, so to speak. So it doesn't matter if you take longer than a natural flier.