My days of long-haul paxing is blissfully over, but back then there was not a single flight when I didn't get up and stretch my legs, walking all the way to the back and up front again. Probably the CC recognized me, so never once was I approached and asked WTF I think I was headed when my head popped through the curtain. Neither was I ever told not to do so. The general rule is, as far as my experience goes, is that you can walk as far aft as you like, but no further forward than your own class of travel. In fact a particularly nice EK CSM, whom I had a chat with in the FWD galley on a 777, wished me good luck on the walk 'down the smelly end', then rung up the AFT galley asking them to have a Jack D ready for my arrival. Sure, I was flying first, and service is somewhat elevated in that class of travel, but still a far cry from 'crew not liking wandering in and out of premium cabins'.
As for the swapping of seats, on the other hand, that's a big no-no. 2 of you have bought not only the seat but also the onboard service that comes with it, and can't just share either with the other 2 who haven't. They're stuck behind the curtains, I'm afraid. If you want to have a chat whilst en route, it's those sitting in comfort who walk back to those in steerage, not the other way around.