EASA has no control over Permit to Fly aircraft, which include much more than microlights. To get an idea of the numbers involved in the UK go here
GINFO Database Search | Aircraft Register | Safety Regulation and enter "PFA" under Serial Number, then do the same for "LAA" and "BMAA". Add to those searches factory built microlights and it shows that non-EASA aviation is alive and well in the UK.