Applying the rocket thrust ahead of the CofG will provide natural static stability, although at high speed in the atmosphere the aerodynamics could well upset that balance.
Apply the thrust below the CofG and you'll need to be able to gimbal it. But the size and weight become a significant factor. Now try balancing a pencil vertically on your hand - I doubt you can react quickly enough to do it. Replace the pencil with a a heavier and longer broomstick and it becomes somewhat easier.