do you really confirm that the rudder and elevator move? For that matter in how many types can't you see the ailerons / spoilers / flaperons from the cockpit?
Some airliners have control position indicators, i.e. A320, B747, B777. Some do not, i.e B737 where you can see the ailerons move but certainly cannot see the elevator or rudder!
A large Austaralian carrier I was seconded to in the 1980's required the Flight crew to carry out a complete visual flight control check whenever any flying control system had been disturbed in maintenance. This was of course in addition to the normal duplicate inspections by LAE's. It took about 5 mins with the F/E observing outside whilst the pilots did the wiggling; I thought that was 5 minutes well spent.
For normal line ops on A/C such as the B737 I do believe it is reasonably safe to assume that if the previous flight was normal ,then with a full and free check of controls, it is safe to assume that the 'invisible' surfaces are actually moving as intended.
Regards
Exeng