I work full time and there's no doubt it takes time. I found taking them at the pace I could handle was the only way. You will always have the people down the club who can fly full time and who seem to pass all 8 exams in a couple of weeks but you need to be practical, and that way the stress levels are kept down.
The only problem you will have is the lack of certain exams, Air Law and Met and Nav may restrict your flying. Stick with it though we're all different.