It took me around 2.5 hours to get from the M23 junction to LHR once - I was pax but it turned out that all the flights were either delayed or cancelled due to thick fog so I ended up waiting on the concourse at T1 for about 3 hours then driving back home again!
Interestingly, once you were about 10 miles from LHR the fog cleared up - I would have been fine if I had chosen to fly from LGW
I also had to sleep on the floor in T1 once due to weather delays - I can't go in there now without a flashback!