You are only allowed to take off under special VFR if the airport is towered or a satellite airport to a Class D,C or Bravo airspace.
That's from my previous post
So if your airport is not towered or a satellite airport, even with E down to the ground, yes you would be in violation.
****** Disclaimer: This is based on my humble PERSONAL interpretation of 14 CFR 91.157 and does not constitute ground instruction nor does it
in any way, shape or form, hinted or implied constitute advise to act accordingly ***********
American litigation laws you know: " But I read it on an Internet forum, so it must be true"