It still might be battery related, try disabling the system battery monitor.
The battery monitor reads /proc/apm, which triggers the kernel to call the APM BIOS. The BIOS then switchs the CPU into System Management Mode while it polls the battery for the amount of charge left. System Management Mode is normally entered with interrupts disabled, so the kernel misses clock timer marks - which causes the real time clock to lose time.