Once you see CHTs in Vy+10 climb on an average Lycoming, you won't even think about climbing at Vy, let alone Vx.
I do it simple: obstacles after TO = Vx, no obstacles = Vy (or whatever required for cooling purposes). I hardly think anyone can object climbing with Vx when there are obstacles present, but there are some reasons why Vy mostly turns out to be better choice for initial climb out.
First of all, as you mentioned, in case of flying at Vx, an engine failure in initial climbout will provide you with a handful of problems. Even with perfect and timely reaction, engine failure at Vx will still require you to push the nose down hard, in order to increase speed to Vg (best glide speed), which is (in most prop aircraft) approximately Vy (give or take couple of knots). Flying with less than Vy brings aircraft to the backside of the drag curve (where drag increases with decreasing speed), which means that you have to be aware of it, otherwise you may easily stall an aircraft while doing memory items during EFATO. You could ask a fellow pilot to pull the throttle to idle at 300ft after takeoff for a simulated EFATO and you'll see the reaction has to be almost instant in order to keep the aircraft flying and accelerating to Vg, while at Vy there is some buffer.
The second problem is lookout. Can you really maintain positive lookout to the front of the aircraft while flying at Vx with some insane pitch attitude or would it be easier to spot an aircraft with smaller pitch (and thus higher airspeed)? Some people try to fly SEPs as jets, but there is too much difference between those type or aircraft to fly them with the same technique. 15° pitch after liftoff will work for most jets, while at the same time is a bad idea for SEP, especially for spam cans.
And the last, as mentioned before - engine cooling. Many engines tend to overheat (CHTs way over 400°F) at sub-Vy speeds due to insufficient airflow through the cooling fins on the cylinders. On hot days, even oil temperature can become a problem at Vy on some installations.