Probably comes from early sailing vessels- port and starboard - in german bachbord and steurerbord (steering side)- The rudder was a steering oar on the starboard side of a vessel - the helmsman stood on the port side and had his back to the port side (bach(k) bord). This meant that the vessel could only dock on the bachbord side to avoid damage to the steering oar - hence port side.
Many naval terms and customs were adopted in aviation.