what affects the ping speed
Probably easier to describe what does not affect ping speed. So many factors.
and is there anything I can intervene with?
Depends on your ping target(s) .... if it's on-net to your ISP then yes, if it's off-net then you would probably be wasting your time moaning to them, especially as a bottom of the pile DSL customer with no SLA/SLGs.
Try accessing web pages hosted on their network and pinging/tracerouting around their network a bit.... it might be that they are having routing issues today.
Even for DSL, that's quite a surprising number !
Not wishing to sound like a call centre agent, but have you powered off your router, waited a few seconds and powered on again ? Sometimes an exchange resync does wonders !