Riding is dangerous and a spooked horse is a risk any rider knowly takes everytime they jump on.
But in this day and age of litigation, it would be wise to make sure that as pilots we are above 500 foot agl in line with the law (unless you have a dispensation for military/police/EMS/SAR etc)