I personally think surface that the horse is travelling on should influence both the uphill and downhill performance of a horse as well as "stability" of the horse.
Riding up or down hill on grass/dirt? Minimal impact on maximum steepness traversable, minimal impact on speed and stamina. Full stability.
Riding up or down hill on mud? medium impact on maximum steepness, heavy impact on speed and stamina. Full stability
Riding up or down hill on "gravel"? BIg impact on max steepness, heavy impact on speed and stamina. Lowest stability.
Riding up or down hill on rock? Big impact on max steepness, low impact on speed and no impact on stamina. low stability.
Riding on flat ground the impact on stamina and speed should still be a thing as well as stability but to a lesser extent.
Riding a horse on uneven terrain like rocks/mountain/gravel should create a massive issue with stability even on flat ground. Stability would impact your ability let go of the horse reins. This meaning the lower stability levels would limit you to be unable to use anything except one handed light weapons(not lances) or to even swing at all. Medium stability levels would make it impossible to swing or couch a lance but also the ability to draw a bow as you need both hands for that, this goes for casting spells as well, one handed spells would be fine though but we dont have any schools for that in game at the moment. High stability levels would just be your typical hill or flat ground "good" surfaces.
Effectively what i'm saying, encourage a mounted player to fight and travel where you have to play to your strengths, giving people on foot sort of "safe" areas against mounted fighters, archers and mages where they would want to try to bring fights. How would a mounted fighter for example fight someone on foot when he starts going up a mountain, or onto lose rocky ground hilly terrain? You either risk it with the extremely limited capability that you could bring there on the horse or you dismount and go on foot. Its also entirely possible to do the completely viable third option, don't pick fights that are to your disadvantage. You are in the same seat the footfighter are when he sees you moving away from the rocky terrain. Should he chase after you into "dangerous" territory and have to fight a mounted fighter in the open? Most likely the person on foot will call himself lucky that he managed to get away and leave the area away from the mounted fighter.
This brings aspects when it comes to travel too, sometimes it might be good to travel on foot due to mountains, snow, gravel, rocks. But most of the time travel is best done on horseback.
With "gravel", I don't mean a path that has gravel on it, but rather natural locations in the world that has rocky ground that has been ground into pebbles. usually underneath and near rocky mountains, close to water, near rocks that someone(not necessarily players, since the surfaces kind of have to be static for gameplay consistency reasons probably) have broken down.
I just think there should be more to horses than them being a scooter that you hop on and only have to worry about its health and stamina and that can travel pretty much just like someone on foot, except faster and with a one-sided damage bonus.
There simply should just be areas that a mounted person will try to want to fight in and there should be areas where they would not want to fight in. So in an engagement there are more than just X vs Y and more X vs Y with Z, Ä. Å, Ö other variables that you have to consider.