You should be OK, MJ, because in any scenario where HMRC determine that a "self employed" person was in reality an employee, it is the case that the alleged "employer" was deemed to be deducting PAYE.
It is therefore the employer who gets done by HMRC for the money they are deemed to have deducted.
This is the standard trap for employers who use long term subcontractors. If the subcontractor status of the person is disallowed by HMRC, the chap can just smile and walk away, and the employer gets done for the deemed deductions.....
At least I hope I understand you correctly (an employer of all kinds of people for over 30 years).