Ph.D. admissions criteria as predictors? (Science Advances)

[robphy]

Possibly interesting reading...

This article ( https://www.inverse.com/article/52691-gre-scores-are-a-bad-predictor-of-phd-success )
alerted me to the article below.
See also: https://phys.org/news/2019-01-gre-students-diversity.html

http://advances.sciencemag.org/content/5/1/eaat7550
http://advances.sciencemag.org/content/5/1/eaat7550/tab-pdf

Typical physics Ph.D. admissions criteria limit access to underrepresented groups but fail to predict doctoral completion
Casey W. Miller, Benjamin M. Zwickl, Julie R. Posselt, Rachel T. Silvestrini and Theodore Hodapp
Science Advances 23 Jan 2019:
Vol. 5, no. 1, eaat7550
DOI: 10.1126/sciadv.aat7550

Abstract
This study aims to understand the effectiveness of typical admissions criteria in identifying students who will complete the Physics Ph.D. Multivariate statistical analysis of roughly one in eight physics Ph.D. students from 2000 to 2010 indicates that the traditional admissions metrics of undergraduate grade point average (GPA) and the Graduate Record Examination (GRE) Quantitative, Verbal, and Physics Subject Tests do not predict completion as effectively admissions committees presume. Significant associations with completion were found for undergraduate GPA in all models and for GRE Quantitative in two of four studied models; GRE Physics and GRE Verbal were not significant in any model. It is notable that completion changed by less than 10% for U.S. physics major test takers scoring in the 10th versus 90th percentile on the Quantitative test. Aside from these limitations in predicting Ph.D. completion overall, overreliance on GRE scores in admissions processes also selects against underrepresented groups.

 

[Vanadium 50]

My understanding of what they have discovered is that after correcting for undergrad GPA and institute quality, there is little to no additional information in the GRE. I don't find this surprising.

 

[Vanadium 50]

Let me make three more comments:

1. This is hard to get right. The definitive experiment doesn't exist. That would be to accept a large cohort of students who have have gone through the usual grad admissions process, but admitted without regard to this process. This cohort would then be tracked to graduation, and only then would the admissions factors be considered in retrospect, to see what would have been predictive. This would need to be nationwide (otherwise at best it is a statement about one or a few universities) and would need to be simultaneously large and small: large enough for the statistical power of the dataset to be predictive and small enough that these entrants would not significantly perturb the graduate school experience.
   Because the data we have is imperfect, how it is interpreted makes a big difference.
2. The authors confuse their own issue by bringing women into the mix. The fraction of women graduating with bachelors degrees in physics, the fraction entering graduate school and the fraction earning PhDs are all about the same, 19 or 20 percent.
3. If there is no additional information in the PGRE, what does it mean for the test to be biased? If you dropped it in favor of higher weight on grades and undergraduate institution quality, wouldn't you be accepting (or rejecting, the argument works in either direction) more or less the same students?
4. I can't count to 3.
5. If you have a threshold, and you determine that how far above your threshold you are has no predictive power, that does not mean that whether you are above or below the threshold has no predictive power.

 

[Dr. Courtney]

First let me say I agree with all of V50's excellent and well-considered comments. Let me add:

1. The study conspicuously neglects any consideration of undergraduate research accomplishments and letters of recommendation, which I regard as at least as important for graduate admissions as QGRE and PGRE scores.

2. The study report cites references 10 and 11 incorrectly on a number of occasions, and even when cited correctly, the findings of these earlier studies do not deserve the level of confidence with which they are asserted.

3. The study assumes that most underrepresentation issues are attributable (and thus fixable) to institutional factors. Lots of high school and college studies show that many underrepresentation issues are more strongly attributable to home life issues: single parents, poor parenting practices, poor home accountability for academic habits, etc. Lots more underrepresented minorities hear that they can be President or doctors than hear that they can be Chemists and Physicists. Underrepresented minorities dreaming of being Chemists or Physicists are likely to be dissuaded from those paths by parents, brothers, relatives, etc.

4. The claim "no study has tested the validity of common admissions metrics [GRE scores] explicitly in these programs" is simply wrong. I think what the authors mean here is that they were unable to find _PUBLISHED_ studies demonstrating the validity across institutions. Thus they make the error that the absence of evidence implies evidence of absence. Many programs internally track their admissions requirements with downstream success rates very carefully and have a good working knowledge of which factors are more likely than others to predict success in THEIR programs. Many know, for example, which admissions criteria correlate best with first year grades, which correlate best with passing the general exams, which correlate best with research productivity, and which correlate with eventual PhDs for THEIR programs. It is foolish to suggest that demonstrating the validity of a given metric across multiple institutions is somehow superior to internally demonstrating its validity at the specific institution at which the admissions decision is actually being made.

5. Over reliance on PGRE and QGRE scores is a necessary consequence of grade gifting at lower tier institutions. Additional objective information is needed to reduce the risks associated with admitting students from lower tier institutions based too strongly on their GPAs. Though I tend to agree with the authors' preference against rigid cut-offs (preferring a holistic evaluation of the total application), I disagree with their implicit suggestion that MORE information is not better than LESS information. Of course, more information is always better, though greater care and personal attention is due to how that information is evaluated.

 

[bhobba]

I think it depends on where you are. In Australia we are moving to Bologna Model. You do a normal degree. If you got a GPA of 5.25 (just over a credit average) in your final year subjects you do a two year Masters of Research. If you get 65% you can do a Masters of Philosophy and when finished do a Phd. If you get 75% then you can go straight to a PhD. No GRE or anything like that - just being able to prove you can do independent research.

Thanks
Bill