imho the hidden agenda doesn't bring as much trouble as the fourth function, the inferior one or the point of least resistance. It can be called the conscious part of hidden agenda. If the hidden agenda is Si, then the inferior function is Se - they are both sensing functions, but of different directions.
For INTJ the conscious need to be good-looking and strong enough, to protect his interests and to persuade people to do something can be really annoying. The point is that you understand you have to apply Se in some situations, but the function is not well-developed, and the experience of performing some Se actions is usually not a pleasant one - especially if they fail. It is in the super-ego block. You understand the need to do something, but don't know how to do it properly. And when there is some information from outside about how to do it, it is not easily accepted.
Hidden agenda is in the super-id block - subconscious. You usually don't think about it and don't perform any actions by the block's functions, but wait for the information from outside. If there is none, you can feel that something's wrong, but not as clearly as with inferior function.