In a society, such thing does not exist but where exactly are the limits?
Not all of us see things in the same light. For example, for people with with high empathy, it would be hard to remain neutral in a situation where a person is likely to get hurt. Would you allow someone to set himself on fire? In theory, it's a matter of personal freedom to do so but something tells me I won't allow it to happen if it was in my power to prevent it.
That example is a bit extreme but what about religion? I don't think belief is harmless because if you trust some external force to fix your problems (for example) then you'll be prevented from attempting to fix them yourself. Like praying instead of giving medical assistance. Also, religions tend to consolidate into political organizations and often such organizations use the political power to do harm. If you support an organization you're accomplice at some level, maybe not directly, but then the supposed neutrality goes away.
I think it's mostly a matter of the capacity of an individual to see long term connections between positions and actions and their consequences. Those who are more inclined to be individualistic tend to be shortsighted when it comes to such consequences, this is, they fail to acknowledge the negative implications of their actions and postures when it's not plain obvious.
I know this area it's been discussed to death but we could apply some Socionics to it.