Armor no matter if evenly distributed or loading weight onto the shoulders and hips, will slow down the wearer and exhaust them. Increased respiratory response from moving while under load is exhausting. Throw travelling in mud, sand, through thick bush or up elevated terrain + rocks into the equation and the exhaustion levels increase significantly due to the body requiring more energy reserves and oxygen to keep up with the muscle fatigue - On top of increased joint and back pressure.
People bearing heavy weight while moving, yes, they can move at their fastest speed - But only for a short burst, before muscle fatigue and respiratory pressure kick in and their speed gradually reduces. Now there's the aspect of combat stresses which not only affect the mental side of things, but physical with cortisol and adrenaline release. These things can make people have longer sustained performance, until things like the adrenaline ware off, and then you get hit with adrenaline fatigue(which sucks btw).
Outside of combat, moving long distances while under load, will affect your performance when the fight comes too, if there's limited rest, hot temperatures etc There's so much that plays into a persons ability during combat while under load. Speaking a lot from personal experience here, not in knights armor or antiquity era armor - But modern plate carriers with up to 40kg+ worth of shit on my back, while in peak Afghan summer.
End of the day, this is a game and the game does need more stamina for 1vX reasons. I do agree heavy armor needs some sort of adjustments, but it would just make it useless unless other areas are tweaked too.