Next time you are looking closely...
It is easy to determine which panels come off regularly on the fleet I fly. I simply note which access panels are normally shiny-new. Typically, it is on the left side of the engine pylon...a circular plate about 6 inches in diameter.
You are correct, shon7, that the panels you describe are designed to withstand flight when their fastening is incomplete. I would not notice vibration on the controls. I am able to feel a very small vibration when the APU inlet door is open.