Some good replies on this thread.
I would just add that have your personal minimums, be also be "generally comfortable" with the minimums that your license permits you to go to.
By "generally comfortable," I mean that if you do find yourself with 3-miles visibility during a flight, you're not getting into a panicked state. If you don't feel you can do this now, go up with an instructor or a suitably comfortable safety pilot.
As some have stated already, never fly when you are not comfortable, but don't ever be afraid to work towards lowering your personal minimums. This is especially true if you want to continue into the Commercial IFR world one day, where your personal minimums have to be at as low as the approved approach minimums for all approach types, at all times.