Well, that's not entirely true.
If you've been denied a medical, you can not legally fly a LSA. Which leads to the bizarre situation that you could have an obese LSA pilot with multiple heart attacks in his past flying on his driver's license, never have applied for a medical, but an ATPL with a failed Class 1 can not.
My point being, if we allow that, then that should apply across the board.