Distortions in Self-Report Data

[Creepy-]

We all know this is a big problem with testing, so I thought maybe we should collaborate and put our insights and ideas into one place for easy access if anyone wants to develop a new test or improve on an old one, etc.

Here's a list of some common distortions (lists and definitions are incredibly useful things).

Social-desirability bias
- a tendency to give "socially approved" answers
- subjects try to create a favourable impression

Misunderstanding
- respondents misunderstand questions
- the way questions are worded can shape participants' responses

Memory errors
(pretty much self-explanatory)

Tendency to agree
- respondent tends to agree with nearly all items

Tendency to disagree
- respondent tends to disagree with nearly all items

There's probably not a great deal one can do about the last three distortions (although a large scale like rmcnew's might help somewhat for the last two).

Social-desirability bias is likely responsible for the "ENTp bias" that SG illustrated here. Maybe test-developers need to make sure that they don't make any functions sound better than others. (I don't have any answer's, I'm just speculating for the moment).

The distortion I'm particularly interested in is the misunderstanding. How does a test developer minimise misunderstanding? Maybe rather than have 1 statement per item, there should be 2 statements which illustrate the same concept in different wordings? Or maybe pairs of positive and negative statements in each item?

Anyway, ideas are wonderful! This is an area that deserves our focus.

[Creepy-]

Bump! I really think this is important.

[Cone]

Haha, I was going to make the exact same reply yesterday, but I thought this topic was still too recent to bump.

I also think it is important and something to consider for future test makers.

[Creepy-]

Haha, I was going to make the exact same reply yesterday, but I thought this topic was still too recent to bump.

I also think it is important and something to consider for future test makers.

I thought it might be too recent as well but I wasn't sure anyone was going to read it in the next few days anyway. I am also going to post (in another thread, for clarity) about Experimenter bias (also important).

[Kim]

I think the only way to work around that is to test for the accuracy of your type in several ways:

1) Take tests AND have people who know you well take the test with you in mind. Compare results. It has worked for me.

2) Read type descriptions and see if they fit

3) Check how the functions work for you

As for tests alone, you can never work around bias in my humble opinion. But one good way really is to be typed by people who know you well.

Maybe test-developers need to make sure that they don't make any functions sound better than others. (I don't have any answer's, I'm just speculating for the moment).

As long as people themselves have a perception of desirable and not-so desirable functions in their head, the way they are presented on the test is not going to make a difference. Especially when they read around here ::

The distortion I'm particularly interested in is the misunderstanding. How does a test developer minimise misunderstanding? Maybe rather than have 1 statement per item, there should be 2 statements which illustrate the same concept in different wordings? Or maybe pairs of positive and negative statements in each item?

Yes, that sounds interesting. But I would have to think about it more