Both could be rather bad...depending
A full rudder hardover certainly has been a real problem in years gone past.
AA had the first on a civil jet, I believe.
A B707 did a half gainer into Jamaica Bay, as a result.
OTOH, if the rudder was completely gone due to a structural failure, control might indeed not be too difficult, provided all engines remained operating.
If the fin departed also, the outcome might not be so pleasant.