It is unlikely that any permanent damage will occur in this situation - although during the event a person might become unconscious but would recover quickly when the aircraft descended to lower altitudes.
Well Mr McGoo, I think you have just found the next area of cost cutting for the airline industry. If we can all survive without the need for supplementary oxygen and then fully recover I guess we really don't need the PSU oxygen unit and drop down masks.