It depends.
If the card is being read or written, you might cause a hiccup. If it's being written, you might lose the data being written.
Otherwise, it's all supposed to be "hot swappable". I've not used the "safely remove hardware" feature in years.