The OP specified a dull punch or chisel was placed against the bearing to provide a point load and that this was then hit with a moderate blow. Just hitting the bearing with a hammer is unlikely to shatter it as the blow is distributed over too wide an area.
Similar principle to hammering a nail into wood. Hit the wood with a moderate blow with a hammer and little or no damage. Concentrate the energy into the point of a nail and it penetrates the wood easily.