Cover your behind.
Call Ops or Duty pilot or Duty manager preferably on a recorded line and express your concerns.
Have them tell you it’s alright.
Board the pax, in case of an incident you should be in the clear.
Unless there was blatant obnoxious behavior in the boarding area I think you jumped the gun.