One reason for the 400 ft altitude limit and line of sight with drone at all times is to keep them clear of aircraft. Pilots use "See and be seen" to avoid other aircraft. Drone operators can not easily judge exactly how the drone altitude compares with a much larger aircraft so must stay below altitudes they normally use. The occasional helicopter that might fly lower can be avoided at ease.
I agree that the rogue operators will always be a problem because they don't follow any rules. Don't let them ruin it for the rest of us.