It's entirely true.
Having been denied a medical does prevent one from flying (legally).
Eliminating the medical entirely means that one doesn't have the medical to stand in the way. My point was simple: I've met many who would fly given the chance, even though they're not physically fit to do so. Presently they're unable, given that the medical stands in their way.
Take away the requirement to hold a medical certificate, and many will put themselves in the air, when they should not be there.
The sport pilot certificate was a step in the wrong direction by effectively accomplishing the same thing.
Presently we have ample individuals who fly when not medically qualified; one who isn't in an airworthy condition isn't allowed to fly, even if one holds a valid medical certificate. A head cold is a good example; I've met many who are more than content to fly with a headcold and sinus troubles, even though the pain from sinus problems can be absolutely debilitating. Holding the medical certificate seems to leave some sense of entitlement among some pilots to fly, regardless of their condition...even if they're not presently medically qualified to do so.
Given that many will fly when they shouldn't, under the mere premise that they hold the certificate (even though it's not valid for flight operations if the individual isn't in good shape), how many more individuals will take flight if no barriers exist?
I've met quite a few individuals who have flown without medical certificates, who disregard the regulation. Having a medical certificate won't stop these individuals, but it may stop a great many others who presently don't fly because they can't pass the physical.
Which leads to the bizarre situation that you could have an obese LSA pilot with multiple heart attacks in his past flying on his driver's license, never have applied for a medical, but an ATPL with a failed Class 1 can not.
A person with multiple heart attacks isn't legal with the driver's license, per 14 CFR 61.23(c)(2)(iv):
(2) A person using a U.S. driver's license to meet the requirements of this paragraph must--
(i) Comply with each restriction and limitation imposed by that person's U.S. driver's license and any judicial or administrative order applying to the operation of a motor vehicle;
(ii) Have been found eligible for the issuance of at least a third-class airman medical certificate at the time of his or her most recent application (if the person has applied for a medical certificate);
(iii) Not have had his or her most recently issued medical certificate (if the person has held a medical certificate) suspended or revoked or most recent Authorization for a Special Issuance of a Medical Certificate withdrawn; and
(iv) Not know or have reason to know of any medical condition that would make that person unable to operate a light-sport aircraft in a safe manner.