Statistics - Within-subjects for categorical data

[JoniGuerrero]

Hello, I am struggling to find a suitable statistic test for one of my studies:
I have 4 different conditions, and I want to test if my results differ on those conditions. When the dependent variable is continuous it is easy to find a statistical test; however when it is a categorical variable I am not finding the right one.

Some details that may help:
Number of users: 23
Categorical Variable: 4 different values (not ordinal.. but can be ordinal if that's the only way)

Any suggestions?

Thanks for the help in advance,
João Guerreiro

 

[Stephen Tashi]

Your description of your goals and your data isn't clear.

For example, it isn't clear whether the same 23 "users", did the same task under 4 different conditions.

It isn't clear what you mean by a "suitable test statistic". What would make the statistic suitable? Statistical procedures are essentially subjective. What are you trying to accomplish? For example, your goal might be to publish an article in a journal. There may be cultural traditions about what kind of statistical results those journals publish. Your choices in statistics would be influenced by that.

 

[JoniGuerrero]

Hello,

Sorry for not being very specific. It's a within subjects experiment, so the 23 users performed the 4 different conditions. What I want is to verify if there are significant differences between the conditions. I have been searching, and I found similar tests, but not one that exactly matches with my needs. For instance the Cochran's Q, which is almost what I want, but only deals with dichotomous variables. Alternatively I can perform the Bowker test of simmetry, but I guess I have to perform it 2x2 at a time (similar to McNemar, but with more than 2 categories).

Are these the best options?

 

[Stephen Tashi]

It still isn't clear if what the subjects did each time was "the same" except for the change in conditions. The nature of your data still isn't clear. You have mentioned 4 different sets of conditions but also 4 categories which are apparently the possible values of the results of each test. What is the format of the data for a single test in a single condition? Is the result one value of the 4 categories? Or is the result a vector of values like a subject might produce if he were asked to classify each of 100 photos in 4 categories such as "Happy", "Sad", "Sentimental","Shocking"?

You mention various statistical tests but you haven't defined what your are trying to do. As I said before, the choice of statistical tests is subjective. Many people who use the jargon of statistics don't understand it. No offense intended, but I naturally suspect this when people want to talk about the "best" way to do statistics but don't seem able to define what they are trying to accomplish.

 

[JoniGuerrero]

It still isn't clear if what the subjects did each time was "the same" except for the change in conditions. The nature of your data still isn't clear. You have mentioned 4 different sets of conditions but also 4 categories which are apparently the possible values of the results of each test. What is the format of the data for a single test in a single condition? Is the result one value of the 4 categories? Or is the result a vector of values like a subject might produce if he were asked to classify each of 100 photos in 4 categories such as "Happy", "Sad", "Sentimental","Shocking"?

You mention various statistical tests but you haven't defined what your are trying to do. As I said before, the choice of statistical tests is subjective. Many people who use the jargon of statistics don't understand it. No offense intended, but I naturally suspect this when people want to talk about the "best" way to do statistics but don't seem able to define what they are trying to accomplish.

Ok, sorry, i'll try to clarify that.
The subjects did the same task in 4 different conditions.
The format of my category variable is just one value of the 4 categories.

 

[Stephen Tashi]

You evaded the important question!

At least the data format is taking shape. Something like:

Possible result = A,B,C,D
Test conditions I,II,III,IV
Subjects 1 through 23


Results Of Testing 23 Subjects:
Subject------Condition---I---II---III---IV
1 -------------------------------A--B---B---A
2.------------------------------ A--A---C---D
3 -------------------------------B--B---C---A
...etc.

 

[JoniGuerrero]

You evaded the important question!

At least the data format is taking shape. Something like:

Possible result = A,B,C,D
Test conditions I,II,III,IV
Subjects 1 through 23


Results Of Testing 23 Subjects:
Subject------Condition---I---II---III---IV
1 -------------------------------A--B---B---A
2.------------------------------ A--A---C---D
3 -------------------------------B--B---C---A
...etc.

Yes, the data format and conditions are exactly like that.

An example similar to the ones I'm performing:
Let's say I asked the users to talk about a person:
The different conditions: That person was a friend (I), an acquaintance (II), a famous person (III) or a specific person I said (IV).
(All users had to perform all these conditions)

The categories would be:
A- He mentioned (how they are related)
B- He mentioned it after some questions
C- He did not mention
D- He mentioned they are not related

And I want to know if there are differences between the different conditions.

Hope it is clear now, and thanks for the help and patience

 

[Stephen Tashi]

And I want to know if there are differences between the different conditions.

Should I take that literally? Or is the goal to persuade someone else about this by writing a report? This is actually an important distinction. For example, if you were writing a thesis that must be approved by a committee, the opinions of the committee about statistical methods also matter.

If it is only your own opinion that matters, we can begin with the fact that you won't know with certainty - at least not from analyzing the empirical data. The best you can get is some statement about the probability that there are differences, and you can only do that if you use Bayesian statistics. If you use traditional statistics, what you get is essentially a statement about the probability of the data given an assumption about how it is generated.

 

[JoniGuerrero]

Should I take that literally? Or is the goal to persuade someone else about this by writing a report? This is actually an important distinction. For example, if you were writing a thesis that must be approved by a committee, the opinions of the committee about statistical methods also matter.

If it is only your own opinion that matters, we can begin with the fact that you won't know with certainty - at least not from analyzing the empirical data. The best you can get is some statement about the probability that there are differences, and you can only do that if you use Bayesian statistics. If you use traditional statistics, what you get is essentially a statement about the probability of the data given an assumption about how it is generated.

Not literally. My goal is to persuade someone else about this by writing a report

 

[Stephen Tashi]

Not literally. My goal is to persuade someone else about this by writing a report

Then we have to consider what would impress the audience. People who are used to frequentist statistics are sometimes vehemently opposed to Bayesian statistics. They also may hold strong opinions about what "p-values" should be used hypothesis tests and various other matters.

Your stated goal is about determining whether there are differences in the different conditions, but you seem reluctant to do pairwise comparisons between conditions. Is this only because of the labor involved? Or are you making a distinction between the statements: "There are differences in the different conditions" and "There is at least one pair of conditions that are different" ? If you published a report that said there are differences in the different conditions, won't readers naturally want to know which pairs of conditions were different?

 

[JoniGuerrero]

Then we have to consider what would impress the audience. People who are used to frequentist statistics are sometimes vehemently opposed to Bayesian statistics. They also may hold strong opinions about what "p-values" should be used hypothesis tests and various other matters.

Your stated goal is about determining whether there are differences in the different conditions, but you seem reluctant to do pairwise comparisons between conditions. Is this only because of the labor involved? Or are you making a distinction between the statements: "There are differences in the different conditions" and "There is at least one pair of conditions that are different" ? If you published a report that said there are differences in the different conditions, won't readers naturally want to know which pairs of conditions were different?

Yes, I also want to show which are the different conditions.
I only discovered Bowker test very recently, and do not see many results using it.. so I would also like to know if it would be adequate to my case (it seems to be the best option). I'm reluctant to do pairwise comparisons because I have to do this kind of test for many different results... and if I had a test for more than 2 conditions I could do post hoc tests only for those that show significance. However, if it is the best option, I have no problem using it.

 

[SW VandeCarr]

Yes, I also want to show which are the different conditions.
I only discovered Bowker test very recently, and do not see many results using it.. so I would also like to know if it would be adequate to my case (it seems to be the best option). I'm reluctant to do pairwise comparisons because I have to do this kind of test for many different results... and if I had a test for more than 2 conditions I could do post hoc tests only for those that show significance. However, if it is the best option, I have no problem using it.

If you're essentially summing over several 2X2 tables for asymmetries, the Bowker test would seem to be appropriate. The result has a chi square distribution. However, to get another interpretation, you can use three individual tables with one group serving as a control.

http://technion.ac.il/docs/sas/stat/chap28/sect26.htm

 

[Stephen Tashi]

All I know about Bowker's test is what I read on the web. Are you thinking of a contigency table like this:

------------------------Condition
Result-----------I-------II-------III--------IV
A-----------------5-------4-------5---------6
B-----------------6-------8------10--------4
C-----------------7-------6--------6--------8
D-----------------5-------5--------2--------5

That lumping together of data could disguise some kinds of differences. For example, suppose people with result A on condition I tended to have result B on condition II and people who had result B on condition I tended to have result A on condition II. Then the totals of A's and B's might not change much, but the conditions are still having different effects. You'll have to decide if you care about that kind of possible difference.

 

[Studiot]

If your report is to carry weight with others you need to present it in conventional terms. That is you need to develop a "test hypothesis".

http://www.google.co.uk/#hl=en&suge...l=&bav=on.2,or.r_gc.r_pw.&fp=a0a3fc0b3fbd3275

go well

 

[JoniGuerrero]

All I know about Bowker's test is what I read on the web. Are you thinking of a contigency table like this:

------------------------Condition
Result-----------I-------II-------III--------IV
A-----------------5-------4-------5---------6
B-----------------6-------8------10--------4
C-----------------7-------6--------6--------8
D-----------------5-------5--------2--------5

That lumping together of data could disguise some kinds of differences. For example, suppose people with result A on condition I tended to have result B on condition II and people who had result B on condition I tended to have result A on condition II. Then the totals of A's and B's might not change much, but the conditions are still having different effects. You'll have to decide if you care about that kind of possible difference.

Yes, it is exactly like that. That's a good point.. the results provided by bowker's test do not provide that kind of differences, which may also be useful. I can perform those tests, and still achieve many of the results i purposed to, but that kind of differences I can't. Do you know an alternative?

Thanks a lot

 

[Stephen Tashi]

Do you know an alternative?

I don't know one off the top of my heaf. I have an hazy intuition about what is needed.
It makes me suspect that the method is already well know. Perhaps it is a so-called "bootstrap" method.

I explain my intuition and perhaps some other forum member will recognize what to do.

To simplify the discussion, pretend there are 3 subjects instead of 23. If we look at the detailed data as:

---------Condition ----I---II---III---IV
subject
1--------------------------A--A--B---C
2--------------------------B--B--C---A
3--------------------------A--A---C---D

Suppose we took the letters that are in each row of the table and wrote them into the same row of a new 3x4 table, but in a random order. A statistic that detects patterns in a table would have a certain average value over all such random re-arrangements. Some such statistics don't change their value (for any given table) if the columns of the table are interchanged or if the rows are interchanged. Let S be that kind of statistic so we will be dealing with something that is independent of the order we choose to write the columns and rows in the tables.

S has some value when computed on the data table and some mean value over random tables. The probability that S falls in | S(data) -mean S) | quantifies the probability of the data given the hypothesis that there is no systematic effect.

The problem is finding one or several suitable statistics! You have to find a statistic that is independent of the order in which rows are written and the order in which the columns are written, but still changes values when you rewrite the letters in one individual row.

Rather than attempt to break new ground in statistical theory, we should first do some web searching to see if people have already worked out the mathematics for this.

 

[Stephen Tashi]

I'll give an example of such a statistic S.

In the data table:
---------Condition ----I---II---III---IV
subject
1--------------------------A--A--B---C
2--------------------------B--B--C---A
3--------------------------A--A---C---D

We notice that there are four A's that visually lie on the vertices of a rectangle. Define S as the total number of such rectangular patterns that appear in the table. Such a rectangle is required to have the same letter at each of its vertices, it needn't always be an 'A'. (I see only 1 such pattern in this example.) The value of S isn't changed by swapping two rows or two columns, only the visual dimensions of the rectangles are. The value of S isn't changed by changing the labels we use for the results.

If we permute the entries of a single row, we may create or destroy rectangular patterns. So S has some variabiity over the set of tables created by randomly permuting each row independently of the other rows.

 

[JoniGuerrero]

Thanks a lot Stephen, you have been a great help. I will search more about that.
Thanks to all that helped :)