No problem in our family. I turned out to be left / right dyslexic and managed a 35 year Airline career and still fly my own aircraft. No. 1 son is and is a Harrier pilot. No 2 is and has 1st Class degree from Oxford and a PPL.
IF you want to fly go for it but be prepared to have to work hard.