Generally, I absolutely agree about always having an 'out'. It's why, in my later and wiser years, I declined to fly single engine over water (cold, cold UK water) out of gliding distance of land. Did it a lot (IOM for instance - 40 miles of Irish Sea from the Lancashire coast) when I was younger. Almost certainly, in a Chipmunk, if the Gipsy fails over the sea you will probably die. No 'out'.
I always did my aeros at a minimum of 3,000 start base so I had room to sort out any mistakes, as well.
However, I wasn't always consistent about this. For instance, I did do a fair bit of low (but legal) flying as it's fun and the view is better. It definitely reduced the choice of fields in the event of engine failure, but it not to the extent that there were none. And of course in busy corridors like the Manchester LLR it vastly reduced the chances of a mid air. So there's a safety balance there.
But one thing I did a couple of times (no more than that) really didn't have an 'out', but only for a few seconds. A 'zoom take off' in the Yak52 where one holds the aeroplane down to gain speed, then translates to a very steep climbout for about 300', before easing down the nose as the stored energy was dissipated. There's a few seconds there where an engine failure would be 'interesting', so it's not something I'd recommend. Which is why I only did it a couple of times.
I did it because it was fun, and the risk of the engine failing in those few seconds was extremely low. Not sure I'd do it now, though, if I was still flying the Yak. And I only ever did it solo so it was always 'my neck' only.
Life is a balance of risk against 'doing stuff'. It's not black and white.