Who knows what's going on inside their head when an emotional situation is in front of them... I try to avoid emotional situations, especially if I find them unnecessary. I bet most are the same way, but they probably would not be comfortable engaging in emotional situations. I really doubt that it's because of being annoyed, it's more like being uncomfortable and not confident in their abilities to handle the situation as they think it's supposed to be handled.
Sounds like your bf has the Fe-polr colored glasses on . When we don't remove those polr glasses we can't see things with clarity.