If you leave the road the "Pet follow speed" and horse Speed could normalize again. Maybe there could be a timer. If you travel on a road for more then 2o seconds the speed buff could kick in.