F test for R^2 when comparing two models

[emmasaunders12]

Hi all

I am comparing the performance of two models and have calculated the coefficient of determination R squared for each. I would like however to test the significance of this value.

The F test however requires that the number of degrees of freedom for each model. The trouble is one model operates in a lower dimensional domain, by projecting the data into PCA domain. Therefore I am not sure on the degrees of freedom for this model as I have degrees of freedom in the PCA domain and then a projection operator to obtain my values of the dependent variable.

Has anyone any ideas how to proceed, in this case, or perhaps the suggestion for an alternative test statistic.

Thanks

Emma

 

[Number Nine]

What are your models, precisely? It seems to me that there is almost certainly a better way to compare them than by R². By the sounds of it, the models have different numbers of parameters, in which case comparing R² values is completely inappropriate. My first thought when I hear "model comparison" is something like the AIC or BIC, but this will depend on what you're interested in (e.g. prediction, model fit, parsimony).

 

[emmasaunders12]

Hi number 9, can I simply alter the degrees of freedom for comparison:

I have found this relationship:

If the models have dierent numbers of parameters, the formula becomes:
F =[(SS1-SS2)/(df1df2)]/SS2-df2

I'm interested how well each model performs, that is how well it fits to new data, outside of a training stage, does that count as prediction?

Thanks

Emma