Just a note but IE on an XP machine is asking for it as MS are no longer updating or patching it meaning the list of vulnerabilities to exploit will only increase.
For further troubleshooting from one of the machine, launch Start>Run>CMD and in the command line window type:
ping 8.8.8.8 -t . This will set up a continuous ping to one of Google's DNS servers. Leave this running, providing outbound pings are not blocked by your router/firewall you should get a continuous reply along with a round trip time like this:
Reply from 8.8.8.8: bytes=32 time=52ms TTL=47
See if there are any "Request timed out" entries in the response over a period of 10-15 minutes. After this period, within the command line window, press CTRL+C you will then get some statistics like this:
Ping statistics for 8.8.8.8:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 52ms, Maximum = 444ms, Average = 154ms
If the above is OK and you get zero loss or something that's less than 2% then there is no packet loss meaning there is no basic connectivity issue.
There is lots you could do here to further troubleshoot and we can try some more things later.
However first of all (long shot) try temporarily specifying an alternative DNS server on one of the computers within TCP/IP settings. Why are we doing this? Well it could be that the first public DNS server your router is hitting is flakey and not responding to all requests to resolve website names all the time. Below is a screenshot.
Specify the examples shown. Now try browsing checking for a consistent experience.