You could argue this like you can argue wearing a drysuit for over water.
Flying alone it's up to you what you wear but in the end it comes down to how "civilised" you want to make your flying with other people. Most people will not want to participate if the barrier is set at this level.
IMHO, a fire (that would go out of control and disable the pilot even if power is turned off promptly) on a well maintained certified aircraft is such a massively unlikely scenario that if you get it, your luck has truly run out.
On some homebuilts, it is less unlikely, and I recall reading of some homebuilts using aluminium fuel pipes to the engine, which is stupid because as the engine vibrates on its rubber mounts, the fuel pipe will break. A few AAIB reports I recall reading more or less said that the thing was on fire and the pilot was overcome by flames while still in the air. But certified types will have a high standard of fuel plumbing, which is not exactly rocket science...