i am an intj guy, and was in a relationship with an esfj girl for nearly 9 months. we broke up because of other circumstances, though not necessarily because we were having major difficulties getting along. what you described sounds almost nothing like my relationship, or even other relationships with different esfj's throughout high school.

my guess is that

a) one of you is not the personality type you think you are
b) he is not physically attracted to you
c) he thinks, for one reason or another, there is no way a relationship between you could work. yes, intj's are less than stellar in relationships, but we are more than willing to work at them, especially if we like the person we are with. on the flipside, we are intelligent enough to know when things just aren't going to work out, in which case you are fucked and would do well to just move on.