I come in from Hemel (near you?) everyday and sometimes the journey off the M25 onto the M4 and into the central area is longer than the trip from Hemel, so I totally see where you're coming from.
Not sure what can be done about it though. There's been many times I wish I could just walk in and out of the central area. Would cut a good 30 mins from my journey time.