There are at least four different reasons for wanting decent visibility, and the minimum that you pick may depend on the circumstances of the flight:
You may need visibility to control the aircraft. If you're not trained in instrument flying, you probably want a decent horizon to help keep straight and level.
You need visibility to avoid other aircraft. That may bother you more if you're heading for a busy GA airfield than if you're flying in a relatively remote area where the chances of encountering another aircraft are slim.
You need visibility to avoid terrain and obstacles. You need better visibility for flying in hills and mountains than you might be happy with in flatland. Seeing a hill a minute before you're about to hit it can be pretty stressful.
You need visibility to navigate (or you may not, depending on the methods of navigation you are trained in and intend to use). Visual navigation in visibilities of even 5 to 10 km can be very hard work. What you decide you need may depend on how difficult the task at hand is.
So while I'm not suggesting that a personal visibility minimum is a bad thing, please remember that different missions need different considerations.