Yeah, it's like some sort of sick joke... On one hand you may like each other a lot, but then you might not be able to work it out when there's a problem.
I've known Se-dominants that seem to have been "attracted" to something about my presence, and not merely in order to cause me pain - i.e. perhaps there was something about me each time that made them feel compelled to get closer.
As for the other way round - and with my knowledge of Socionics - it feels to me like a naughty pleasure. I don't mean that I intentionally pleasure myself with such people - I mean that it seems like a frivolous interchange where I may get burnt or where I may make a brief connection with the individual, perhaps a moment of mutual understanding. But generally (not just with conflictors), I feel wary, like I'm going to be caught out, so often, any physical attraction I am at such a distance that attraction as a result of a person's physique is not possible.