I've never had a problem either (travelling by GA) but have had to show the passport in quite a lot of places, as described.
Luggage is something else. At many places, one ends up in the airline passenger queue as the only option. The x-ray people have to go through the motions but have to let it go, whatever they see in the backpack in the way of sharp tools, etc. You get accompanied by some staff member, having used the words "private pilot" several times. The only place where I actually had stuff confiscated was Norwich airport where they confiscated my toothpaste, and the b*stards even had a big laugh about it. Even at Tirana, where I got considerable hassle, they didn't do that.