QNH1013, have you ever actually been formally audited for IR35 compliance?
I would say that as an FI(R) it is impossible to legally self employed.
When I was a engineering contractor and enquired if my part time instructional work could be invoiced though my umbrella company then the answer was no.
As an FI it is possible to be self employed, but check your paperwork carefully.
Several individuals have worked on a self employed or Ltd basis for several years, only five years later find themselves facing an audit to confirm the self employed status and find themselves facing a bill for additional tax.