What is the issue with calling INTps contrary?

I think both types are fairly idiosyncratic, but for different reasons.

INTps seem more contrary to me, but maybe that's because I am INTj