Despite being overweight the CofG was withing spec (which I'd say is more important).
Well yes, but. For aircraft handling but there are times when performance matters... a lot. High density altitude operations come to mind. Short runways. Obstacle clearance. All those nice charts in the manual for T.O distance over a 50 ft. obstacle suppose that the aircraft will deliver specified rate of climb. Reduce that rate of climb an you can find yourself in dangerous territory. I had a sphincter-tightening experience in my previous Cherokee 140 once. There were three of us; I'm fairly heavy at 200 lb, but my two passengers were 140 and 125 lb respectively (both pilots). We were just a bit under MGTOW; field was 2500 ft, soft (sand and gravel), and had obstacles. And it was at about 2000' above sea level. And it was a hot day, about 30C. We cleared the treetops by a very uncomfortable margin.
It was one of those "I learned about flying" experiences that you'd read about in Flying magazine. I knew, roughly that at MGTOW 2500 ft was OK for the Cherokee, but I hadn't thought that only 2000 ft above sea level there would be such a performance drop. When I got home I plugged the numbers into the charts (I know: should have done that first. I now know better). Well, I was surprised to learn... the available field length/obstacle clearance distance was right on the limits for density altitude, and the soft field correction.
And that wasn't even
above the maximum.
I now own a Beech Sundowner 180, and it has considerably more payload. but also, a greater fuel capacity (with two markings on the tabs for partial fuel load computation) so it requires more careful planning. At sea level, standard conditions, max all-up weight, the distance to clear a 50' obstacle is 1860 ft. I frequently fly to a 1600 ft paved strip in the Gulf of St. Lawrence. Beautiful spot. Fortunately no immediate obstacles, and usually a good stiff headwind, but it is short enough to concentrate the mind, and to make me ensure that I know I am within the performance envelope.
I also once owned a C150. Great little aircraft but with one big flaw: the ability to land in a distance way too short for takeoff! With two chaps on board, climb rate was nothing to brag about.
So, yes, to an extent CofG can be more dangerous, especially exceeding the rear limit. Assuming you can safely get it into the air, as you will have considerably less performance on take-off if you're overweight.