Its an endless circular argument but fascinating nonetheless.
It comes down to an equation of risk vs reward:
Is the risk of operating with reduced automation more than offset by the benefit of maintaining a high standard of instrument flying skills?
I like poling these things around as much as anyone but I have come to the conclusion that no, with the new generation of aircraft its not justified.
My experience is that a basic level of manual flying ability is retained and available in the highly unlikely freak occurrence of loss of all autopilots, after all, every take-off and most landings are still flown manually.
With manual flying and visual apps I think you either have do it alot, as I did doing domestic flying or military, or you pretty much avoid it as on long haul you are never going to retain the level of proficiency to make it as safe as using the automation.