I passenger a lot too. For this reason I do not listen to the safety briefing because I know it by hard. As do a lot of other people on board. But I also do know where the emergency exit is. Always have a look while boarding.
Who cares if you know it by heart? Do you ignore the takeoff briefing because you know it by heart, too?
The flight attendant as a qualified crewmember is providing a briefing...and you as a qualified crewmember ought to be paying attention. Whether you're part of that crew or not.
Personally, I remove the briefing card, open it up, and give the flight attendant my full attention during the briefing. It's good practice regardless of who is watching, and a good example to those around you.