Yeah the system doesn't seem to work that well.
Either you toggle PvP on and get murders for self-defense actions or you don't toggle and can't help anyone.
I feel they need to add another layer to their criminal system, where if you actually see a criminal attack someone (certain radius around you or something like that) their name changes color and you can attack them freely.
That way you could retain the idea of not everyone automatically knowing who's a crim but still have it not mess up with defending against groups of crims attacking your town etc.