This is actually why I don't use a shield on my back as a mage, and opt to keep my fists out for that maximum movement speed. There is no delay on casting if you have your empty fists out. Unless I'm out alone. Then I bring a dagger and shield so that I can defend myself.