It sounds unlikely, but the wording of a contract is deemed of low importance of HMRC decide to investigate you. They use the duck test, if it walks like, quacks like etc.
If your work pattern looks like employment (hours dictated by employer, operating practice dictated by employer, no freedom to accept other roles for other airless etc etc) then HMRC can decide you are an employee and go after you if you have not paid tax and NI on that basis. So if what you are doing looks like conventional employment you should be signed up as an employee, to protect yourself.
You never, ever (no matter how many years later this happens) want to be on the wrong side on a HMRC investigation.