The Numbers Moved

If you learned selection science any time in the last thirty years, you learned a rank order, and the rank order was wrong.

It went like this: general mental ability is the best predictor of job performance, at a validity around .51, and everything else is a supporting act. Work samples, structured interviews, integrity tests — all real, all useful, all somewhere below the cognitive test that sat at the top of the table. That table came out of the field's proudest achievement — Frank Schmidt and John Hunter's decades of meta-analysis, the work that showed validity generalizes and freed employers from re-validating every tool in every building. Generations of I-O psychologists memorized the ordering. It shows up in textbooks, on certification exams, in vendor decks to this day.

In 2021 a team led by Paul Sackett pulled the table apart and rebuilt it, and the top of it is not what you memorized.¹

They say the cognitive test is king

The old ranking wasn't fabricated. It was the honest output of a correction — and the correction was slightly too aggressive, in a way that took twenty years to see.

Here's the mechanics, in plain terms. A validity coefficient measured in the real world is attenuated: range restriction (you only have outcome data on people you hired) and criterion unreliability (performance ratings are noisy) both drag the observed number down. So the field corrects for them — legitimately — to estimate what the validity would be in the population. The problem Sackett and colleagues identified is that one of those corrections, the range-restriction one, had been applied with an assumed correction factor that was far too large, inflating the corrected validities of exactly the predictors that get used early in the funnel, cognitive ability chief among them. The tools got credit for a restriction that was more assumed than measured.

Redo the corrections with defensible assumptions and the numbers move. General mental ability comes down substantially from its .51 perch. And the predictor that rises to the top of the operational-validity table — best validity, and one of the better validity-to-adverse-impact tradeoffs in the whole kit — is the structured interview.²

Why this is not academic

It would be easy to file this under "researchers argue about decimals." It isn't, for two reasons.

The first is that people made real decisions on the old numbers. If you built a selection system that front-loaded a cognitive test because it was "the best predictor," you optimized for a ranking that no longer holds — and you very likely bought the adverse-impact problem that comes with cognitive testing in exchange for a validity edge that has since shrunk. The tradeoff you accepted was priced wrong.

The second is subtler and it's the reason this belongs in a magazine and not just a journal. A lot of practitioners are carrying a number in their head — .51, cognitive ability, top of the table — and citing it with the full confidence of settled science. It is the confident citation that's the hazard. A wrong fact you hold tentatively gets updated; a wrong fact you hold as bedrock gets defended. The most dangerous sentence in an analytics review is "the research says," delivered about research the speaker last read fifteen years ago.

The pattern worth naming

Call it a frozen prior: a finding that was correct when you learned it, that you promoted from current estimate to permanent fact, and then stopped rechecking. The validity table is a frozen prior. So is half of what any of us "knows" about a field that keeps moving.

The fix isn't to distrust the evidence — the meta-analytic tradition that produced both the old table and its correction is the most trustworthy thing selection science has. The fix is to hold the numbers the way the people who produce them do: as the current best estimate, dated, and subject to revision when someone checks the correction factors again. Quote the .51 if you must, but quote it with its expiration: that was the number before we found the range-restriction assumption was too aggressive; the corrected picture puts the structured interview on top.

The rank order moved. If yours hasn't, that's not rigor. It's a citation you stopped reading.

---

Companion piece: Borrowed Validity — on why even a correct meta-analytic number is a starting estimate, not a local verdict.

Footnotes

1. Sackett, Zhang, Berry & Lievens (2021), Journal of Applied Psychology — a meta-analytic re-analysis correcting decades of over-correction in range-restriction adjustments; 234+ citations and field-consensus-shifting. Applied implications in Sackett et al. (2023), Industrial & Organizational Psychology. ↩

2. Per the corrected operational-validity estimates, structured interviews rank at or near the top on validity and offer one of the better validity–diversity tradeoffs among widely used procedures. The pre-correction ".51 for GMA" figures trace to the Schmidt–Hunter validity-generalization corpus and should no longer be quoted as current. ↩