Your reasoning is quite correct, and you've almost answered your own question!
Va is nothing to do with the rudders, it is the speed at which a pull to the limit load factor (2.5g for public transport a/c) will result in a stall. A speed less than this, and it you will stall before you reach the limit load factor, above this speed it is possible to overstress the aircraft.
The important thing is that Va relates to the limit load FACTOR, and not the actual load on the structure. eg) 2.5g for public transport. At a lower weight, the stall speed is lower. Therefore, you can pull more 'g' before you stall. This means that Va must be lower (the speed at which a pull of 2.5g just results in a stall).
The flight envelope Va is calculated for the maximum design weight. It is interesting to note that the at a lower weight, the load on the structure will be less at the limit load factor. eg) the weight of the aircraft times the limit load factor is less. Therefore, the Va to give the same limit load FACTOR decreases with weight, but the Va to give the same limit LOAD (in Kg) is independent of weight.
Complicated I know, but hope this helps.
