(imho) it's often because the "appropriation" is centered on the cheesiest stereotypes of a certain culture i.e. the "weird dress" of indians, the "mafia and pizza" for italians, etc. etc. thus people feel offended that you're conflating their culture with a single and usually somewhat negative-cheesy trait.