Wheelybin, you are entirely correct in that the reply should be "descend to altitude 5000'". However, you are NEVER told to descend by a set amount (for instance, in the hold at 9000', you are never told "descend 1000'" to mean "descend to altitude 8000'" Therefore, there can never be any real confusion here? I always use the word feet, to readback that I understand its an altitude and not a FL.
The only real danger is if the clearance is 'descend two four zero zero (2400) feet' can be misinterpreted as descend to 400' (as claimed the lives of a cargo 747 a while ago.