Many sport aviation types as well as gliders have exemptions from both to operate in Class E. They have done for many years, and regularly operate in Class E.
Presumably ATC would have no awareness of such aircraft, if they can't see them and/or can't talk to them.
If the airspace is upgraded to E because of increasing pax movements / traffic density, then why should ultralight aircraft have exemptions from complying with the requirements of class E airspace (in particular radio / transponder fitment). There is plenty of class G airspace in Australia, and LSA / Gliders can go no radio, no transponder, no SAR till their hearts content. But there is NO place for LSA no radio / no transponder aircraft at aerodromes like Ballina with multiple Jet movements.
If these exemptions are going to be pulled, the SAAA/RAA etc. might have something to say.
They can say all they like, perhaps they could start with why they think it is ok to operate in the same airspace as multiple high capacity jets (as is the case in the Ballina area) with no radio and no transponder.