Seeking advice on creating ratio scales from ordinal scale instruments

[thelema418]

I have a psychological testing instrument that produces an ordinal measure (0, 1, 2, 3, 4, 5). I want to change this to a ratio scale with range 0 to 5.

The instrument is designed so the first 5 questions are very easy (Level 1), the next 5 questions are harder (Level 2), the next set is even harder (Level 3)... etc. A participant scores 0 if they cannot answer any questions. They reach the "Level" if they answer 4 out of 5 of any numbers at that level.

To make this a ratio scale, I was thinking about scoring with the following points per problem at each level:

Level 1: Each problem worth 1 point
Level 2: Each problem worth 6 points
Level 3: Each problem worth 36 points
Level 4: Each problem worth 216 points
Level 5: Each problem worth 1296 points

I was going to take the sum, $x$, and then calculate $\log_6{x}$.

I'm wondering if there are any other suggestions about how to do this.

 

[Simon Bridge]

There are infinite ways to turn thisinto a ratio scale.
i.e. just score each question as shown ignoring the levels, add the scores and multiply by 5/<max possible score>.
So you need to spell out the constraints on the problem - what is the proportional score supposed to represent? How will it be used?

 

[thelema418]

The constraint is that I want to keep this feature built into the instrument: "The instrument is designed so the first 5 questions are very easy (Level 1), the next 5 questions are harder (Level 2), the next set is even harder (Level 3)... etc. A participant scores 0 if they cannot answer any questions. They reach the "Level" if they answer 4 out of 5 of any numbers at that level."

A level 1 question is more like a drawing of 3 circles and asking how many circles are there. A level 5 question is like creating a proof by contradiction with awareness of non-Euclidean mathematics. So there has to be a weighting on the problems.

The scale needs to reflect the differences in the difficulty of the problem sets.

 

[Simon Bridge]

The scheme in post #2 reflects the difficulty of each level by giving a bigger score to the higher level questions.
All I have basically said is there is no need to take a logarithm.

The details of the weighting applied dependson what you want the result to tell you - what are you trying to achieve?

After all, if all you want to do is preserve the "level" feature then you'd just be using the old system.

 

[thelema418]

I don't understand what you mean by "just score each question as shown ignoring the levels" (#2). If you added an image in that post, it is not showing.

 

[Simon Bridge]

Score as follows:
Level 1: Each problem worth 1 point
Level 2: Each problem worth 6 points
Level 3: Each problem worth 36 points
Level 4: Each problem worth 216 points
Level 5: Each problem worth 1296 points

add up the points for all correct answers, divide by 1555.
This will give you a score out of 5, which is weighted according to the difficulty of the questions, as requested.

 

[thelema418]

Ok, that makes more sense. It might give me some normality issues, but it makes sense. thanks

 

[Simon Bridge]

If you made each question at level n worth 10^n-1 points, then the raw score will also preserve the achievement level. Divide by 11111 to get an overall score out of 5.

Overall, what you use depends on what you want to do with it.
You will need to check the scoring approach against some data to see if the scores follow a normal distribution.
Considering the way the questions are ranked, I would not expect it to anyway.