I think with both flying, and riding a bike, the handling skills do stay with you.
But the ability to simultaneously handle, navigate, communicate, monitor what the aeroplane's doing, follow checks, and maintain situational awareness - all at once, that's what goes. So, stuff gets missed - and that I think is what tends to make a less-current pilot relatively dangerous. Mostly, they can still take-off and land okay.
Much the same with a bicycle - when I was a student who went everywhere by bike, I was quite safe and comfortable cycling places on a busy main road in all weather - my handling skills and SA were sharp enough that I made it to 25! Now, I'd be lucky to survive the experience intact and my very occasional cycle rides I constrain to quiet country roads.
G