To start with, the primary objective of flying Vx is for obstacle clearance. That is to say; Gain as much height at the least distance. This is achieved at the cost of the following:
1. Slower IAS, thus that much closer to the Vs.
2. Climbing at a higher AoA, thus that much "buffet margin" in a Bank.
3. Climbing at a higher AoA, thus that much less forward visibility.
4. Climbing at slower IAS. thus that much less of kinetic energy.
If, for argument sake, you know that your engine is going to fail at X second after lift off, then if you fly at Vx, you would have gained more altitude that you could have gained at Vy speed, however, being that much closer to Vs, a slight delay in Engine Failure recognition will jeopardize your IAS and perhaps the height gained by Climbing at Vx is insufficient for a successful recovery.
If stall is averted, then you will have lost that much height that will force you to take anything straight ahead anyway, so the advantage of climbing at Vx is therefore gone.
Climbing at that much higher AoA as to gain Vx, will leave you with that much less buffet margin especially during a High Angle Bank that might be required for a turn back to the runway. The risk to stall your aircraft during the turn back is that much greater, because of less airspeed and a higher AoA.
While you may know your area very well, by having less forward visibility, that reduces your chances of finding out if something is suitable for a straight head forced landing, as this is a lot less dangerous than a turn back.
Arguably, during a Low Level engine failure scenario, your kinetic energy will do you more favor by giving that much margin to the above mentioned risk, potential energy (i.e height) does not have significant advantage (because you are low anyway) while poses more risk by having the above mentioned problems.
So, I suggest, unless your specific type of aircraft mention otherwise, Vy will give you more margins for an EFATO.
Safe Flight,
SS