I learnt as a sailor that port was the side that traditional ships were designed to bring to dock. So I guess that the "steering board" would have been on the other side makes sense. Port has the same number of letters as left. I also learnt the port (wine) / red colour association for the lights. I understand that the channel markers are the other way around in the US, but either way it depends whether you are going upstream or downstream as to which you hold to your left.
I've only been a pilot for a couple of years but have never heard the use of port/starboard in an aviation context or textbook. I must admit I did wonder, but left/right seems to work OK. I haven't seen it mentioned in any of the old flying books I've read either, such as Stick and Rudder or Sagittarius Rising.
In sailing we called "Starboard" to announce our intent to exercise right of way due to the tack we were on. (Pissed off a couple of bigger and considerably more expensive boats a couple of times who were apparently working on the assumption that size and wealth were the deciding factor).