Your oath of loyalty is to the institution of the monarchy, to the Crown, not to the particular individual who is wearing it at the moment.
We have been very fortunate in our recent monarchs. Perhaps Charles will not attract the same admiration as QE2, but that is because she ascended while very young so we know nothing of her views and opinions. Charles has been a busy and concerned PoW who has ruffled feathers but will have to go silent when he wears the crown. I am sure he will due his duty well.