Irreversible controls can still move in the circumstances described.
irreversible refers to the fact that the controls do not move in response to changed aerodynamic loads on the surface - so if I put the elevator in position A it stays there, until I decide to move it again, even if the aerodynamic hinge moment chnges, say due to a speed change.
But - unless the flight controls themselves are perfectly balanced, the changes in loads in the control system due to, for example, 'g' oscillations can cause an input to the controls.
Oh - and not all aircraft will stabilise hands-off, of course. Neutral stability in the Phugoid mode isn't that unusual, and isn't really a bad thing (low drag tends to reduce Phugoid stability, and low drag is generally a good thing). The Phugoid mode is sufficiently slow that it can be easily controlled even if mildly unstable.
For specific examples: I know of one case with an "irreversible" pitch control curcuit where, due to an unbalanced control system (a design oversight, it turned out to be), there was 'g' feedback to the controls and the aircraft was PIO-susceptible as a result. We had to redesign elements of the control system as a result.