Are you able to monitor the signal strength and link quality that the WiFi adaptor is getting?
Even with low levels of both measures I'd be surprised if the wifi link was limiting your ADSL speeds.
What tool are you using to measure? You need to use more than one to get ajn accurate read, so I'd suggest the speed test tool at
http://www.adslguide.org, and using
http://www.broadbandspeedtest.net/