From an aerodynamic point of view, you're probably right. However, you're also responsible for the welfare of the fare paying passengers in the back, who are quite rightly much more interested in their own comfort than Mr Boeing's lovely wing.
If the turbulence is less than severe, why would you not choose a speed which makes the ride more bearable for those behind you without the 5 point harness?