O-P, what you assert is correct in a vacuum - but not otherwise.
Take a piece of A4 paper and drop it out of the window. Now take an A4 piece of steel and repeat the experiment - I think that you'll find that the time taken to reach the ground is rather less for the latter case.
Two forces are acting on the object, weight (mg) and drag (Cd½ρV²S); the resultant force which accelerates the body is (weight-drag) and only weight has a mass term. As the body accelerates, drag increases until it is equal to weight, at which point there is no further acceleration and the body is descending at terminal velocity.