- #1

JPierce

- 6

- 0

I have a project where I am analyzing the results of multiple reviewers on a set of items. I am unsure as to the proper method of normalizing the data.

In essence, here is the problem:

We have a large stack of assignments turned in by students, with one assignment turned in by each student. Teachers analyzed each assignment according to three criteria, which I will call A, B, and C. Teachers values for each of these criteria on a scale from 1 to 5. However, this scale is not linear, so A=4 is not twice as "big" as A=2.

I simply want to display a summary of the results using (say) a histogram. I have no interest in calculating summary statistics because the numerical values for each criterion are purely denumerable -- that is, A = 2.4 (which could correspond to say grade level) is meaningless.

So far, so good. But some of the assignments were reviewed by up to five teachers. Others were reviewed by only one.

So one assignment turned in by (say) Jimmy may have the following reviews from five individual teachers:

A = 3; B = 1; C = 4

A = 3; B = 2; C = 4

A = 3; B = 1; C = 2

A = 2; B = 3; C = 4

A = 3; B = 2; C = 4

Another assignment turned in by Mary may only have A = 3; B = 1; C = 4 as measured by a single teacher.

So, how do we handle the fact that some assignments have more reviews than others? We could just scale up the number of reviews to a common value. In other words, we could pretend that Mary turned in five identical assignments, that is,

A = 3; B = 1; C = 4

A = 3; B = 1; C = 4

A = 3; B = 1; C = 4

A = 3; B = 1; C = 4

A = 3; B = 1; C = 4

Somehow that doesn't seem quite right. And it would foul up the precision of the results.

Another idea is to whittle down the number of reviewers to 1 for each assignment, but I have no good criteria for selecting the one sample to keep.

Any ideas?

If I have left out important info, just let me know.