Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

F test for R^2 when comparing two models

  1. Mar 11, 2014 #1
    Hi all

    I am comparing the performance of two models and have calculated the coefficient of determination R squared for each. I would like however to test the significance of this value.

    The F test however requires that the number of degrees of freedom for each model. The trouble is one model operates in a lower dimensional domain, by projecting the data into PCA domain. Therefore I am not sure on the degrees of freedom for this model as I have degrees of freedom in the PCA domain and then a projection operator to obtain my values of the dependent variable.

    Has anyone any ideas how to proceed, in this case, or perhaps the suggestion for an alternative test statistic.

    Thanks

    Emma
     
  2. jcsd
  3. Mar 11, 2014 #2
    What are your models, precisely? It seems to me that there is almost certainly a better way to compare them than by R2. By the sounds of it, the models have different numbers of parameters, in which case comparing R2 values is completely inappropriate. My first thought when I hear "model comparison" is something like the AIC or BIC, but this will depend on what you're interested in (e.g. prediction, model fit, parsimony).
     
  4. Mar 11, 2014 #3
    Hi number 9, can I simply alter the degrees of freedom for comparison:

    I have found this relationship:

    If the models have di erent numbers of parameters, the formula becomes:
    F =[(SS1-SS2)/(df1df2)]/SS2-df2

    I'm interested how well each model performs, that is how well it fits to new data, outside of a training stage, does that count as prediction?

    Thanks

    Emma
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook