The issue with walls is that it makes your house/village more safe than a town is. This means that everyone moved for extraction, etc. to their own villages and left the towns.
Due to walls you weren't able to see any other payers in the world, and the world felt dead and empty, even through the...