I would actually say it's more realistic that there are no handle hits on a 1h sword. It would be incredibly rare to strike someone with the butt of your weapon while intending to go for a blade hit IRL. A sword is like all blade for the most part. The level of ineptitude needed to actually strike someone with the butt of your weapon without intending to is staggering. Even just sparring in your backyard with no formal training, it's not too hard to keep the pointy end facing your opponent.
So yes. I don't think handle hits should be a thing on 1h swords. The fact it is feels super gamey and not realistic at all. And clearly, it isn't needed for balance reasons. I'd also agree with increasing the blade strike area on 1h axes and hammers. While handle hits are more realistic for these weapons, like swords, when you actually hold them in your hands its a bit easier to control them going where you want than in a video game where you can't really feel the weapon in your hands.