50ft originated in the early days of US military aviation when Curtiss provided a demonstration at an Army base (the name of which eludes my memory searches). The parade ground from which the demonstrations occurred was surrounded by trees approximately 50 ft high. There were more than a few finger in the wind rules raised in the early days to start the regulatory ball rolling.
Source - course about 25 years ago at which one of the instructors was a rather aged ex-CAB engineer and who was a boy in the office at or about the time.
35ft, as I recall from conversations with somewhat older certification folks, came about in the early days of jets when 50ft was found to be too limiting and a bit of equivalent safety horse trading was the salvation.