I suspect that it's going to be, because the association has been successfully made between them and terrorists. And it's a similar scenario to when someone says "I'm not a pedophile" or "I'm not a nazi." The denial might be true, but they'll assume you are one just because you deny it. "Oh, I'm not a terrorist - " no terrorists would admit to being one. But now, the association is between weak men, and terrorism. So it is a dangerous term in that regard.
That's the logic. Like publicly calling someone "nazi" or "pedophile," it's a damaging charge even if it turns out to be false.