Of course, I'm not suggesting anything other than "good terms", ie: I'm not saying you should actively snub him. Just that your contact will likely continue be at best clearly superficial and as IEIs, we both know that's really unsatisfying and empty.
I would tend to feel like I'm "pretending" and eventually the emotional buildup would probably generate a scene in yer classic
sense.
I'm sure he doesn't want to give up your friendship. But with his wife threatened by you, he's preserving his marriage. He sounds like a decent ethical guy. Perhaps the situation will change eventually. Or he'll just get tired of the insecurity. IME, people eventually get tired of being mistrusted. The wife obviously doesn't really trust him, let alone you.
I wish you well. It's a real bind. <3