Overboost is N1/EPR higher than the max rating for the ambient conditions (i.e. above Takeoff or Max Con, depending on altitude/airspeed). In Normal mode, overboost is prevented by the EEC.
In Alternate mode, the EEC doesn't have enough validated data to calculate the ratings (N1 engines) and/or can't validate EPR (EPR engines). Although the specifics change between engine types, it's possible to significantly exceed the max rating for the flight conditions in Alternate mode (i.e. overboost the engine). Overboost is primarily an economic concern - the engine will rapidly deteriorate, but it's generally not unsafe.
Overspeed means you've exceeded the redline value for a rotor speed (N1, N2, and for Rolls N3). The EEC will always protect rotor speed redlines in both Normal and Alternate mode. Aside from a momentary transient overshoot, if the rotor speed exceeds redline, something in the engine control has failed. Exceeding redline is potentially unsafe and can result in an uncontained engine failure.
Again, it varies between engine types, but all engines have some sort of backup overspeed protection to protect against potential engine control failures (e.g. control is lost of the fuel metering valve and it goes wide open). Some engines use an electronic system that, if it senses an overspeed it'll shut the engine down, others use a hydromechanical system (essentially a fancy fly-ball governor) that will bypass fuel to maintain rotor speed at redline. In most cases, the overspeed protection system only protects the high rotor (N2 or N3) - the engine manufacture has to show that low rotor will maintain structural integrity for the worst case low rotor speed with the high rotor at redline.
In general, the only difference between Normal and Alternate mode is that Alternate mode is "unrated" and overboost is possible. In some cases the engine control will use somewhat more conservative control algorithms to account for the uncertainty in ambient conditions, which can result in a slight increase in fuel burn, but the key word is "slight".