Ed, wouldn't you want the aircraft to respond to your control inputs if you were flying it? If the pilot's stick inputs were to not be responded to or obeyed, don't you think there would be a greater cause for concern?
FBW or not, if I pull back on the stick, I sincerely hope the aircraft responds with an appropriate elevator command to do what I asked it to do.
(Granted, in normal mode, it seems that sometimes the aircraft doesn't do this ... different topic.)