Yes, it may be a problem. Are you aware that faulty PBs kept YELLOW PTUs running?
Aside from that, IIRC the ECAM action for a G or Y hyd loss doesn't say turn off the PTU! And of course, in the event of a G or Y reservoir low quantity (or overheat) the PTU p/b will be shining away with a lovely 'FAULT' light. The PTU isn't FAULTy though and will be happily pumping the other system into a reservoir overheat. Moral of the story, know your systems.
Or maybe add some more logic to the PTU controller and remove the button altogether...?