It's a human factors problem. The problem is that the signage does not match the words. As a supposed human factors educated industry, it is pretty embarrassing really.
All seat belts fastened signs should really be "safe to leave your seat" signs. Seat belts should be mandatory at all times. When the sign is on you can't leave your seat. When it is off, then you might stand up to use the lav or stretch your legs.
But currently we are telling people one thing and showing them another. Unfortunately, airlines don't do anything if it costs even 1/100th of a penny. So this needs to be regulation driven.