Yes it can be a problem and it will get worse. Some companies it is actively discouraged as you are no doubt aware.
I've always maintained, the best hand flying practice for honing basic IF skills was to hand fly the aircraft in the cruise, raw data. If you can do that with a reasonable amount of accuracy, then you are doing OK. However since RVSM does not allow this, the next best thing is fly a raw data approach. I am also one who will go for the A/P disconnect button, earlier rather than later, if things are not as expected with the automatics.
There are schools of thought however that while one is hand flying, it is effectively reducing his/her lookout monitoring etc., thereby increasing the load on the other or rest of the crew members. In some situations, this is true, especially as they are all 2 crew cockpits now. depends what sort of a job of hand flying is being made I guess.
Had a young person new on type, in fact any type, recently on a dead leg, asking if he could hand fly it for a period. Yes sure I replied, as I actively encourage hand flying, when conditions are suitable. However passing through the mid 20's on the climb out on this occasion, the head F/A came up front and asked whats wrong? Nothing I said, why? We are all feeling sick back here, the aircraft is going up and down she said.
Confirming the A/P does a better job than most of us humans.