I've never flown a glider, but pax'd several times.
I'd say there's still a fair amount different between gliding and powered flight. If you could get a few hours concession for your glider hours, I'd still think there's a lot of value in going (quickly) through the GFPT sylabus, for your own safety as much as anything. You will wizz through some sections, but like any stude, somethings like going around, for example, need to be practiced to be relatively automatic in order to work well - I doubt you've done many of those in a glider!
Most people have got their stick-and-rudder stuff well sorted (if not fully polished) by the GFPT stage and between GFPT and PPL is largely navigation stuff anyway. I can't see you saving much time on this section.
IMHO your gliding will be a fantastic background, but the GFPT/PPL training is good fun and worth doing for the sake of it.
Good luck - go and enjoy it!