If you are not happy turn back.
And if anyone gives you gip tell them to **** off.
You are PIC you make the calls. If your not happy bin out.
Its all very well discussing it in the bar but if your sphinxter starts twitching and you have a get out clause. Trust it and fly again another day.
Good choice, big lumps of hard stuff in front off you unable to remain VFR you made the right choice.
MJ