Just because something is possible does not mean it is without additional risk.
Minimum use heights for lesser autoflight categories might be because of reliance on an assumed level of human action to recover from failure; the system might fail 'hard' due to a lowered levels of redundancy, internal monitoring, etc.
The height limits are established during flight test, they are often judgements of expected human performance. If they are abused - 'it's ok because we are visual' doesn't guard against the nature of surprise or having a nasty bump.
Similarly for human limitations in lower visibilities; operational limits must be respected as they are designed to cover a wide range of expected scenarios and different pilots' ability. You don't know where your rating stands on a particular day or in the local conditions.