I mean, you're contesting a farming spot so you should expect danger from other players and just because there is guard tower nearby doesn't mean you're invicible and should be able to free farm risk free.
Also, why would you ever dismount and wait for them even if by guard tower, sounds to me...