I think not. By flying people, we are using our hard earned skills to give passenger an experience that they cannot achieve by themselves. So it is not unreasonable to ask for payment.
I would suggest that once you
ask for payment from a non-pilot you are getting close to the realms of charging for services - technically a commercial operation for which you require a CPL. If they
offer payment then you are limited to the cost sharing rules as defined in the ANO.
I think the answer's clear - bite the bullet, enjoy your flying and if someone offers you something for it then accept graciously. Explaining how much it costs and how much you
expect them to pay is putting you on very dodgy ground legally. I've flown with many people who've never asked me for payment, I've flown others and have never asked myself - personally I think it's an appalling abuse of your licence privileges if you are asking and I'm sure the CAA's legal department would be interested too.
CM