I'm only a lowly Engineer here but when I was looking into doing my PPL licence last year, I understood that as long as you only charge for what what it costs you (I.E fuel and landing fees etc) and don't actually make a profit out of it then there wouldn't be a problem.
However, another implication I guess would be the insurance aspect if something were to happen to one of your pax during one of these pleasure flights.
This is only my understanding of it, somone else feel free to correct me if I'm wrong (Which is more often than not!)
Hope this helps