If it's a permanent magnet alternator with no separate field windings then the regulator has to either:
1/ shunt the excess output, or
2/ chop (to varying degrees) the A/C waveform to limit the output. The former was common with British motorbikes 20-30 years ago, and was done with a LARGE Zener diode mounted on a heatsink. Wasteful but simple. The second approach would need more circuitry and would give rise to considerable harmonics on the output, hence the capacitor to reduce this.
The Rotax 2 stoke engines can be fitted with regulators to provide DC battery charging, but 2 types are available, the cheaper version needs a minimum load to work correctly. It may be that your regulator is not capable of switching off sufficiently to keep the voltage within limits unless, as you seem to have found, a small load is applied. A basic magneto equipped aircraft, unlike a car, doesn't need much power to run its systems, so this could explain the voltage rise after a few minutes. Indeed with a conventional regulated alternator it is quite possible for the field to switch off for several seconds if there is no load on the electrical system.