Validation
Check us — don’t trust us
Measures produced by this software end up in dissertations, published validation studies, and operational decisions. So the question that matters isn’t “does it have a Wright map?” — it’s “if a reviewer challenges my numbers, can I point at evidence this engine is correct?” This page is that evidence, and the checks below run automatically against every change to the engine.
Reference calibrations
Every dataset below has an external answer key — a published calibration, an archived analysis, or an independent estimator. Logit must reproduce each one within documented tolerances for the test suite to pass.
A real instrument-development study
Head-to-head vs WINSTEPSFive career-assessment subscales · YouGov panel, n = 500 · Rating Scale Model
Reference: The study team's archived WINSTEPS 4.4.6 output (2019)
Result: r = 1.0000 against every archived item table; mean |Δ| ≈ 0.012 logits — within the rounding of WINSTEPS' own printed output
Liking for Science
Head-to-head vs WINSTEPS · the textbook RSM exampleWright & Masters (1982), Rating Scale Analysis · 75 children × 25 items, 3 categories
Reference: WINSTEPS' published Example 0 item calibration (winsteps.com)
Result: All 25 published item measures reproduced: r = 1.0000, mean |Δ| = 0.04 logits against the two-decimal published table. (Winsteps does not publish the category thresholds for this example, so those are regression-pinned, not oracle-checked.)
LSAT Section 6
Independent third engineBock & Lieberman (1970) · 1,000 examinees × 5 items, dichotomous
Reference: The ltm package's published constrained-Rasch calibration (Rizopoulos, 2006, J. Stat. Software)
Result: r = 0.99999 against an independent marginal-ML engine; centered differences within the documented MML↔JMLE/CMLE metric tolerance
Knox Cube Test
Head-to-head vs a published textbook calibrationWright & Stone (1979), Best Test Design · 35 persons × 18 items, dichotomous
Reference: The book's published UCON item calibration (Tables 3.5.1 & 5.2.1)
Result: All 14 calibrated items reproduced within ~0.06 logits: r > 0.999, mean |Δ| < 0.07 across a 9-logit spread. UCON is joint ML — the same estimator family as our JMLE — so this is a direct head-to-head.
Cross-engine agreement
Internal consistencyeRm reference datasets · dichotomous, rating scale, and partial credit
Reference: Logit's two estimators checked against each other: TAM (JMLE) vs eRm (CMLE)
Result: Item measures agree at r = 1.0000 (dichotomous, RSM; |Δ| < 0.05 centered) and r > 0.999 (PCM)
Known, documented divergences
Two correct programs can disagree in the last decimal for knowable reasons. We document these rather than hide them — a footnoted divergence is credible; an unexplained one is disqualifying.
- JMLE scale factor (~0.9–1.0)
- WINSTEPS applies its own finite-sample bias handling to JMLE; TAM's differs slightly. This appears as a small uniform scale factor on the logit metric, never as item-level disagreement — orderings and spacings match. Fitting the same model (the true RSM) removes almost all of it.
- MML vs JMLE/CMLE metric
- Marginal-ML engines (like ltm) fix the person distribution at N(0,1), while conditional/joint ML estimate the person metric from the data. Absolute logits differ by a small scale; centered calibrations agree.
- Published rounding
- WINSTEPS tables print two decimals; our tolerances respect that rather than claiming false precision beyond it.
What’s not verified yet
Honesty cuts both ways. Residual-based dimensionality diagnostics (principal-components analysis of residuals) are on the roadmap and will arrive with simulation-recovery studies, since no single published answer exists for every configuration. Some of WINSTEPS’ more specialized tables aren’t implemented yet. New psychometric features don’t ship without a reference check or a recovery study behind them.
Methods & versions
Estimation runs in R via TAM (joint maximum likelihood; true Rating Scale Model via a shared-threshold design matrix) and eRm (conditional maximum likelihood). Comparisons use centered item measures (both programs center items at 0 unless noted). The hosted engine builds on the rocker/r-ver 4.4 image with the current CRAN releases of TAM, eRm, and ltm installed at build time, and the checks on this page run in CI against the same current-CRAN package set on every commit — a release that breaks any of them cannot ship.