They should make it so, that you auto-stow your weapon to cast a spell, but that you have manually draw your weapon if you have finished your cast.
I would actually really enjoy this. Quite annoying that my character draws and stows their weapon in between a charge and a cast even if my plan is to immediately blast the enemy in the face with the charged spell.