F test for R^2 when comparing two models

  • Context: Graduate 
  • Thread starter Thread starter emmasaunders12
  • Start date Start date
  • Tags Tags
    Models Test
Click For Summary
SUMMARY

The discussion focuses on comparing the performance of two statistical models using the coefficient of determination (R²) and the F test. The challenge arises from one model operating in a lower-dimensional PCA domain, complicating the calculation of degrees of freedom. Participants suggest that comparing R² values is inappropriate due to differing parameter counts and recommend alternative metrics such as AIC or BIC for model comparison. A formula for the F statistic is provided for models with different parameters, emphasizing the need for careful consideration of model fit and prediction accuracy.

PREREQUISITES
  • Understanding of R² (coefficient of determination)
  • Knowledge of PCA (Principal Component Analysis)
  • F test and degrees of freedom in statistical modeling
  • Familiarity with AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion)
NEXT STEPS
  • Research the application of the F test for model comparison in R
  • Learn how to calculate AIC and BIC for different statistical models
  • Explore the implications of PCA on model performance evaluation
  • Investigate alternative model comparison techniques beyond R²
USEFUL FOR

Statisticians, data scientists, and researchers involved in model evaluation and comparison, particularly those working with PCA and regression analysis.

emmasaunders12
Messages
43
Reaction score
0
Hi all

I am comparing the performance of two models and have calculated the coefficient of determination R squared for each. I would like however to test the significance of this value.

The F test however requires that the number of degrees of freedom for each model. The trouble is one model operates in a lower dimensional domain, by projecting the data into PCA domain. Therefore I am not sure on the degrees of freedom for this model as I have degrees of freedom in the PCA domain and then a projection operator to obtain my values of the dependent variable.

Has anyone any ideas how to proceed, in this case, or perhaps the suggestion for an alternative test statistic.

Thanks

Emma
 
Physics news on Phys.org
What are your models, precisely? It seems to me that there is almost certainly a better way to compare them than by R2. By the sounds of it, the models have different numbers of parameters, in which case comparing R2 values is completely inappropriate. My first thought when I hear "model comparison" is something like the AIC or BIC, but this will depend on what you're interested in (e.g. prediction, model fit, parsimony).
 
Hi number 9, can I simply alter the degrees of freedom for comparison:

I have found this relationship:

If the models have di erent numbers of parameters, the formula becomes:
F =[(SS1-SS2)/(df1df2)]/SS2-df2

I'm interested how well each model performs, that is how well it fits to new data, outside of a training stage, does that count as prediction?

Thanks

Emma
 

Similar threads

Graduate F-test regression test, when and how?
  • · Replies 5 ·
Replies
5
Views
2K
Undergrad Linear regression and random variables
  • · Replies 30 ·
2
Replies
30
Views
5K
Graduate Testing regression model with F-test
  • · Replies 2 ·
Replies
2
Views
3K
Issues with SLS Model for Rubber and Foam Compression Tests
  • · Replies 5 ·
Replies
5
Views
982
High School Chi-squared test stat choosing a critical value
  • · Replies 5 ·
Replies
5
Views
4K
Undergrad About chi-squared and r-squared test for fitting data
  • · Replies 5 ·
Replies
5
Views
9K
MHB Performance of students - Hypothesis testing
  • · Replies 20 ·
Replies
20
Views
4K
Undergrad Questioning the timescape model
  • · Replies 27 ·
Replies
27
Views
2K
Graduate How to correctly analyze the results of a preference study?
  • · Replies 13 ·
Replies
13
Views
2K
Undergrad Degrees of freedom for t-test for 2 samples, 2 variances
  • · Replies 7 ·
Replies
7
Views
6K