Because of your wiring problem your signal strength (Signal/Noise ratio) was probably so low it was only just working during the day, It would not affect your download speed enormously so you wouldn't notice it much.
At peak times when more people log on your signal strength will be reduced even further (because of your contention ratio) and obviously to the point of your modem deciding it wasn't enough of a signal to recognise.
If its working fine plugged into the test point you must try to find out what is wrong with your wiring, or if your happy to leave it plugged in to the test point just leave it there.
I had a BT engineer telling me that last week he had a major fault with about 100 customers who kept losing broadband in the evenings, the fault turned out to be from a single customers modem that was putting noise on the line of everyone else via the exchange, a faulty modem. But if yours is fine in the test point it cant be that.