There are some problems with this 3 Rem number:
1. Rem is not the best unit to measure the effects of radiation exposure. The effect on the human body depends not just on the dose, but also on the type of radiation absorbed. During lunar missions, the astronaut is exposed to an unusual type of radiation - extremely energetic particles from outside the solar system, whose biological effect is not well known, because the only way to get exposed to such radiation on earth is to put your head in a particle accelerator. There are reports of flashes seen by astronauts with their eyes closed, the cause of such flashes is unknown but the most probable cause are those high energy cosmic rays. Radiation that is able to create flashes in your eyes, can't be good for your health...
2. The exact exposure is not predictable. An unexpected spike in solar activity can increase the exposure many times. If there were just a few Apollo missions and they reached 3 Rem - it is reasonable to assume that during hundreds or thousands of missions, the maximum exposure will be several times higher.
3. The effect of "small" radiation doses is the subject of debate. No one can say for certain that a certain dose is "safe".
EDIT: here is some evidence that the Apollo missions were not harmless for the astronauts health...
"At least 39 former astronauts have suffered some form of cataracts after flying in space, according to a 2001 study by Francis Cucinotta of NASA's Johnson Space Center (see journal references below). Of those 39 astronauts, 36 had flown on high-radiation missions such as the Apollo Moon landings. Some cataracts appeared as soon as 4 or 5 years after the mission, but others took 10 or more years to manifest."
Blinding Flashes - NASA Science