The first commercial passenger aircraft I flew was a turbo prop that had no A/P and no F/D.
In fact I remember during the ground school for my next type, our instructor talking about the 'flight director'. We thought 'why is he going on about the guy in head office?'! What he was saying didn't make sense, but nobody said anything so muggins here eventually puts his hand up "errm.....sorry, what's a flight director?"
And as RAT 5 says, this could be the crux of the problem. On that first type I developed a true scan: and even today when I glance at the PFD I look at the speed tape, the pitch, the bank, the altimeter tape and the V/S tape. I don't really look at the F/D much if the A/P is in - there is no point, because even if the square is centered in the cross, the A/P and F/D might have both gone wrong and it would still look right.
I like the idea of flying with EITHER A/P or F/D but not both. On Airbus, switching both F/Ds off forces the A/THR into speed mode, so I don't know if this would be practical for all phases of flight though?