I had that feeling of trepidation when hours building in Florida however, it does wear off in time. Believe the term used is "complacency".
Seriously now, you can always lighten your stressload by tuning into the relevant ATC frequency regardless of whether the station is open or not. When a station is closed give a position report and request traffic information so that anyone near you may pass on any pertinent information regarding their whereabouts.
Keeping a good scan going obviously shows good airmanship but you are perhaps giving yourself too much stress over it. The chances of a midair collision even in the heaviest of populated skies are rather slim, especially if you're making use of all those ATC options available such as flight information and radar information.
Chill out a little but keep that scan going and remember that no relative velocity equates to possible danger.
VFE.