Tip speed: as the prop tip becomes transonic it loses efficiency. As you have noted larger props turn more slowly, but this is mostly a design limit imposed by tip speed.
Clearance: If a prop gets very big, you start to have to design around it such as extending the gear (more weight, cabin entry considerations) or are forced into a high wing multi-engine (which is slightly less efficient)
Regarding piston engines: most props are bolted directly to the crankshaft to avoid the complications, weight and reliability issues associated with a gearbox. In this case the engine RPM will basically determine the radius of the prop, and the engine horsepower will determine the number of blades required to "absorb the power". But too many blades is not good.
Turbo-props have to have a gearbox to reduce the turbine RPM, so the prop size is mostly determined by clearance issues. RPM are generally in the range of about 1000, although it can be significantly higher and somewhat lower.
When you go to a fan design, you get rid of the problems associated with the blade tip. A larger fan can increase bypass ratio and efficiency, but it adds to weight and you need the space to mount it.
I am not qualified to discuss unducted fans or helicopters!
I hope that helps.