You are better off building your territory in the wild where you can defend without consequences.
If they had made the whole world outside of the influence of towns be wild then most of this issue would be gone.
That is what I was hoping they was planning with the zones. But it is not the way they went.