Its the only answer you can have.
You can't refuse access to class D because you say you can't trust VFR traffic.
There is no way VFR traffic will be more trust worthy unless there are a few public cases go through so that level of traffic get there **** together.
And letting a danger to air safety go and do there stuff in someone else airspace/ cause mayhem in class G doesn't solve the problem.