I'm sure you could go smaller if you had the technology or inclination but what would be the point? Everyone knows that with engines bigger is better!
What about - get more power out of given displacement and weight?
The stresses in moving parts depend on acceleration. Which means that if your individual pistons are smaller and have smaller stroke, you could run the engine at a faster rpm, and get more power out of a large number of smaller cylinders than you could have got from fewer and individually larger cylinders.
The biggest infernal combustion engine produces 115 000 horsepowers, and has 14 cylinders (in a straight row). But the largest mass produced plane with infernal combustion engines had IIRC something like 26 000 horsepowers, produced by 168 pistons distributed between 6 engines. Presumably for a reason.