- #1

- 19

- 0

Hi everyone!

This is an application question. I would like to get some advice about how to calculate a score based on a set of individual scores in a way that makes most sense.

CONTEXT:

I am going over some criteria for judging usability of hypotheses. I came up with a whole bunch about a certain topic and now I'm trying to select the best ones. I asked a number of people to evaluate these hypotheses on the criteria below.

A) hypothesis is specific

B) hypothesis is verifiable

C) hypothesis has a strong theoretical foundation

D) hypothesis can be tested using available resources

...

Let's say people evaluated these on a 10 point scale with 10 being the best.

I want one score based on all four criteria above. The easiest way would be to just add the mean individual scores. For example, if average ratings of a given hypothesis were 7,8,9, and 10 for criteria A, B, C, and D respectively, then the hypothesis would get the score of 34. But I wonder if addition would make sense. Here are some potential challenges:

1) if a hypothesis cannot be tested using available resources (criterion 4), then no matter how highly I evaluate points 1-3, I cannot use it. Such hypothesis could score higher than an alternative which was evaluated less highly on criteria A-D, but highly on criterion 4.

2) some of the criteria are highly corelated with each other. For example criteria 2 and 3

will be more highly correlated with each other than criteria B and C.

3) even though there may be a nearly perfect correlation between some criteria across the different hypotheses, conceptually these are different. So, averaging scores based on highly correlated criteria would not make sense.

How would you address the challenges above?

What are better alternative ways of obtaining a single score (alternative to A+B+C+D)?

I will greatly appreciate your help!

This is an application question. I would like to get some advice about how to calculate a score based on a set of individual scores in a way that makes most sense.

CONTEXT:

I am going over some criteria for judging usability of hypotheses. I came up with a whole bunch about a certain topic and now I'm trying to select the best ones. I asked a number of people to evaluate these hypotheses on the criteria below.

A) hypothesis is specific

B) hypothesis is verifiable

C) hypothesis has a strong theoretical foundation

D) hypothesis can be tested using available resources

...

Let's say people evaluated these on a 10 point scale with 10 being the best.

I want one score based on all four criteria above. The easiest way would be to just add the mean individual scores. For example, if average ratings of a given hypothesis were 7,8,9, and 10 for criteria A, B, C, and D respectively, then the hypothesis would get the score of 34. But I wonder if addition would make sense. Here are some potential challenges:

1) if a hypothesis cannot be tested using available resources (criterion 4), then no matter how highly I evaluate points 1-3, I cannot use it. Such hypothesis could score higher than an alternative which was evaluated less highly on criteria A-D, but highly on criterion 4.

2) some of the criteria are highly corelated with each other. For example criteria 2 and 3

will be more highly correlated with each other than criteria B and C.

3) even though there may be a nearly perfect correlation between some criteria across the different hypotheses, conceptually these are different. So, averaging scores based on highly correlated criteria would not make sense.

How would you address the challenges above?

What are better alternative ways of obtaining a single score (alternative to A+B+C+D)?

I will greatly appreciate your help!

Last edited by a moderator: