I think the (left and right) edges of the envelope are the places where you run out of elevator authority at slow speed: The elevator hits the stops before the required (certified) aerodynamic effect is reached. Those stops are literally that - hard stops, so there's no real gradient at the edges. You either hit the stops trying to achieve what you want to achieve, or you don't. Although there will be a safety margin, of course.
As far as flying characteristics is concerned within the envelope, obviously with higher weight comes lower performance (take-off distance, rate of climb and landing distance mostly, although there's also an effect on cruise efficiency, and the speed at which the best glide is found).
Move the weight fore and aft and you'll find that with a forward CofG the aircraft is very speed stable (it recovers quicker from speed excursions), and with an aft CofG the aircraft is more efficient: It cruises faster for a given power setting. The latter is explained because the tailplane doesn't have to work as hard keeping the tail down, which means less induced drag from the tailplane.
And then there's the effect on morale from rear-seat passengers. Depending on who you bring, this can be positive or negative.