Inter-Rater Reliability Study - Need Volunteers

[somavision]

Hello

I have decided to perform a few correlational studies into socionics, the first of which will be a general investigation into inter-rater reliability. This is in part a response to Niffweeds analysis of his "me too" and more specifically a study into what traits are observed and typings given in the absence of the "me too" variables.

I am collecting a wide source of anonymous autobiographical and biographical material which will be issued to each of the coders. The coders will be given time to study the material and given a questionnaire. Whilst the content of this questionnaire is still to be finalised, it will be something along the lines as follows

1. Which IM do you think is most likely the individuals base function
2. How confident are you of this (scale 1-5)
3. Which temperament do you think is most likely
4. How confident are you of this (scale 1-5)
5. What quadra do you think is most likely.
6. How confident are you of this (scale 1-5)
7. What type do you think is most likely
8. How confident are you of this (scale 1-5)
- repeated for each set of sample data
9. How would you rate you own knowledge and understanding of socionic theory (scale 1-5)
10. How would you rate your own ability to code a persons type from the information (scale 1-5)
11. Please add any further information you wish to use to qualify your answers

All data will be treated in strict confidence and the results will be published without usernames. Once the data has been published, the original data tying it to the raters usernames will be destroyd. However there will be an additional question on the questionnaire which wil include the names of those who have agreed to take part.

12.
a. How would you assess coder A's ability to rate their own aptitude in coding anothers type (1-5, Don't know)
b. How would you assess coder B's ability to rate their own aptitude in coding anothers type (1-5, Don't know)

etc.

This information will be published seperately. I will do a statistical analysis and I would also like a volunteer do do an independent analysis, however apart from the data from the last question, all data will be published in full (minus user names)

As such I am looking for volunteers to code the information, the more the merrier. You don't need to be an expert, as this is not the point of the study, but you should be able to honestly assess yourself.

I am also looking for someone to do an independent statistical analysis, this person will receive the raw data, minus the user names. This person must agree to keep all information confidential apart from that which has been stated.

Please pm me if you are interested in either of these roles. The list of coders/participants will be finalised by Monday 7th May and instructions sent shortly after.

I also welcome suggestions with regards to the design of this study e.g. how much information you require to type someone, format, question wording etc.
All participants have the right to withdraw at anytime (as such all data submitted will be destroyed) and there will also be a debrief.

If this study is a success I have a number of experimental studies that I would like to follow up with.

Thank you for reading and I hope to hear back from as many of you as possible.

[mu4]

If you want Soma, I can install a mod for this. I've been meaning to install a form mod for Vbulletin which will allow for the integration of form into the site.

This will get the info into the mySQL and I can provide that data in a standardized and anonymized format to anyone who needs it. Let me know. Also another forum can be setup too for this which can be wiped.

http://www.vbulletin.org/forum/showthread.php?t=201097

[somavision]

If you want Soma, I can install a mod for this. I've been meaning to install a form mod for Vbulletin which will allow for the integration of form into the site.

This will get the info into the mySQL and I can provide that data in a standardized and anonymized format to anyone who needs it. Let me know. Also another forum can be setup too for this which can be wiped.

http://www.vbulletin.org/forum/showthread.php?t=201097

That would be great cheers. The only concerns I would have is that by having the forms come into me directly before they've been anonymised I can keep an eye on trolling/doubling up, also it would allow me to use the data where the coders do the pre-rating of the other raters.

However, I don't think that this aspect is really necessary and I would be happy if the information was sent to me in an anonymised format.

So yes, please that would be great, thanks.

[mu4]

Soma, I've installed the mod, you should see a forms button at the top. I have a form in there that I have created. I will get you permission to create and make this form within the editor, it's pretty simple, or I can help you do it.

There are quite a few options with the mod and it has a export to excel feature as well which can then be anonyomized. Once the polling is done, the form can be deleted and the results purged as well..

[somavision]

Soma, I've installed the mod, you should see a forms button at the top. I have a form in there that I have created. I will get you permission to create and make this form within the editor, it's pretty simple, or I can help you do it.

There are quite a few options with the mod and it has a export to excel feature as well which can then be anonyomized. Once the polling is done, the form can be deleted and the results purged as well..

That's terrific Hkkmr, I can't see the button though, is this just me being a bit blind? I can try to muddle through, but any assistance would be greatly appreciated.

Cheers

[mu4]

You should have access to the form list and test form now.

[somavision]

HI, bit of a delay on getting this thing set up.

I've had two volunteers so far, ideally I could do with three more, so if anyone else is interested, please let me know.

hkkmr, thanks for setting up the forms app, however one of the participants is not on the forum, so I think it might be easier if I either do the forms in a mixed format or use some other method.

EDIT: actually scratch that - I've seen the forms tab, but don't get how to create a form, could you help with this?

Thanks

[tcaudilllg]

interrater reliability has no meaning in a context that treats truth as opinion.

[somavision]

interrater reliability has no meaning in a context that treats truth as opinion.

Given the variables I intend to look at, I am inclined to disagree with you.

Please expand.

[mu4]

Soma, I've started a form. It looks like there would need to be multiple forms, 1 per user that wish to be assessed.

[RSV3]

How is this study progressing?

[somavision]

How is this study progressing?

lol, it's not at the moment, other things came up and I've not had enough people express an interest, but I will try to pick this or something similar up again, maybe nearer the autumn.