Penguina,
I am like you. Very cautious indeed. I will quite happily fly with lots cloud and 8k viz, with a decent x-wind, etc, etc. But not when the base is 2000' or lower ('cept in circuit) or any adverse effects present within 100 miles!
I try to not ask anybody what they think of the conditions. I have noticed suprise from some when I say "not going" and I feel a little silly too when it turns out fine. Then one day when at the club I said "its flyable" one of the instructors asked to come as he was bored. Half way around the circuit the vis reduced to about <1km and I lost sight of the runway, it was IMC by any other name. Was I glad to have him with me! I vaguely noticed a dark line, thinking this was the runway I lined up on it. My passenger noticed the actual runway was to the left - I lined up on a hedge!
I am convinced he knew what was in store but was too decent and wise to question my decision directly. One of my more startling learning moments!
I think stick to your guns and fly only when you feel is comfortable. Ignore this dodging black cloud derring-do and cock a snoot at suggestions of sub standard training