You have a point. Although I think it's not only an Airbus problem. Lots of non-Airbus pilots are also taught to fly without actually flying the plane. But this is more of a company policy problem.
Luckily I work for a company where there's no restriction on manual flight, and although we only fly with FBW aircraft (A320), we fly manually and raw data very often with lot's of visual approaches.
I can only compare with the B737 from my experience in the sim during my MCC, and in my opinion the only difference in flying both aircraft manually is the auto-trim on Airbus.
Just trying to say that you can be a button pusher in any other plane as well, not only Airbus.