It may not be dependent on the engine, but on the ability of the rotor to produce thrust.
At high altitude, the blade pitch angle and hence angle of attack will be quite high, and the increase in drag for an increase in angle will greatly outweigh the increase in lift.
Since drag has to be overcome with power, with a rapidly increasing drag, the engine may not be able to do anything to increase the lift.
For example, the OH-58, at maximum weight, can't make it much over 16,000' pressure altitude at +5C (it gets hot here in Mojave). At lighter weight, it can go a lot higher, but if you work out the coefficient of thrust, it is the same number (weight over (air density times disk area times Rotor tip speed squared - the formulae is in most helicopter engineering books).
Even going above continuous power, at 16,000' +5C, it won't climb at 50 KIAS.
That's why the flight manual will have some different numbers for service ceiling (or should, if it has it at all)