I would have thought that any non-returned ballot papers can only be counted as "Abstentions", i.e. they are not bothered either way. I would say, though, that they cannot be confirmed as accepting any ballot, but then again, they cannot be accepted as rejecting it either.
I would suggest that you can only take the vote from those that return their ballot papers. Anything other way of counting only falsifies the result as I see it.