Item Response Theory
Table of Contents
Item Response Theory explains the relationship between an individual’s ability and their observed performance (Columbia University Mailman School of Public Health, n.d.). Formally, the probability of correctly answering an item is a function of the person’s ability and visualized as the item response curve.
A person’s “ability” and an item’s parameters are organized on an unobservable continuum (Columbia University Mailman School of Public Health, n.d.). The main purpose of IRT is to establish an individual’s position on that continuum (Columbia University Mailman School of Public Health, n.d.).
Glossary
| Term | Definition |
|---|---|
| $i$ | subject $i$ |
| $j$ | item $j$ |
| $N$ | number of subjects |
| $M$ | number of items |
| $a_j$ | (item) discrimination parameter |
| $b_j$ | (item) difficulty parameter |
| $c_j$ | (item) guessing parameter; the chance of a correct response for a very low $\theta$ |
| $d_j$ | probability of endorsing an (item) incorrect answer in spite of high ability |
| $D=1.7$ | scaling constant |
| $\theta_i$ | (subject) ability. Formally cited as $\theta \in (-\infty, \infty)$, it more often is the case $\theta \in [-3, +3]$. An estimated ability of 1.2 can be interpreted as 1.2 standard deviations above the average ability in the population. |
| $\mathbf{r}_{ij}$ | response from subject $i$ to item $j$ |
Assumptions
- Monotonicity: As the trait level “increases,” the probability of a correct response increases.
- Unidimensionality: There is one dominant latent trait being measured and this trait is the driving force for the responses observed for each item in the measure.
- Local independence: Responses given to separate items in a test are mutually independent given a certain level of ability.
- Invariance: We can estimate item parameters from any position on the item response curve. Put another way, invariance means that item parameters do not depend on the characteristics of a particular population. Parameter invariance is crucial if one wants to carefully assess the degree of inferential generalizability across examinee populations; if parameters are not invariant, the statistical foundation for inferences is not identical across populations and hence the inferences are not generalizable across those to the same degree (Rupp and Zumbo, 2006).
Models
Unidimensional, Dichotomous
| Parameter | Formula |
|---|---|
| 1PL | $\mathbb{P}[\mathbf{r}_{ij}=1 | \theta_i, b_j] = \frac{\exp(D(\theta-b_j))}{1+\exp(Da(\theta-b_j))} = \frac{1}{1+\exp[-Da(\theta_i-b_j)]}$ |
| 2PL | $\mathbb{P}[\mathbf{r}_{ij}=1 | \theta_i, a_j, b_j] = \frac{\exp(\theta_i-b_j)}{1+\exp[a_j(\theta_i-b_j)]} = \frac{1}{1+\exp[-Da_j(\theta_i-b_j)]}$ |
| 3PL | $\mathbb{P}[\mathbf{r}_{ij}=1 | \theta_i, a_j, b_j, c_j] = c_j + (1-c_j)\frac{1}{1+\exp[-Da_j(\theta_i-b_j)]}$ |
| 4PL | $\mathbb{P}[\mathbf{r}_{ij}=1 | \theta_i, a_j, b_j, c_j, d_j] = c_j + (d_j-c_j)\frac{1}{1+\exp[-Da_j(\theta_i-b_j)]}$ |
Optimization
Maximum likelihood estimation picks the most likely ability and item parameters given the observed responses. Subject and item parameters $\theta_i, a_j, b_j, c_j, d_j$ are optimized with gradient descent. While MLE is simple to understand and implement, it lacks any measure of uncertainty and can have important consequences when responses are missing.
\[\mathcal{L}_{\text{MLE}} = \max_{\{\theta_i\}_{i=1}^N, \{a_i, b_j, c_j, d_j\}_{j=1}^M} \sum_{i=1}^N \sum_{j=1}^M \log \mathbb{P}(\mathbf{r}_{ij}|\theta_i, a_j, b_j, c_j, d_j)\] \[\theta_{i} \leftarrow \theta_i + \eta \frac{\partial\mathcal{L}_{\text{MLE}}}{\partial\theta_i}\] \[a_j \leftarrow a_j + \eta\frac{\partial \mathcal{L}_{\text{MLE}}}{\partial a_j}, b_j \leftarrow b_j + \dots, c_j \leftarrow c_j + \dots, d_j \leftarrow d_j + \dots\]Expectation-Maximization
Hamiltonian Monte-Carlo
Wu et al. (2020) proposed VIBO, a variational inference method for fitting IRT models to large datasets. Applying this method to five large-scale item response datasets from cognitive science and education yields higher log likelihoods and improvements in imputing missing data.
The authors made the assumptions listed below
Assume:
$\theta_i \in \mathbb{R}^{K} : \mathbb{P}[\theta_i] = \prod_{k=1}^K \mathbb{P}[\theta_{i,k}]$ where $\mathbb{P}[\theta_{i,k}] \sim \mathcal{N}(0,1)$
$\mathbb{P}[\{a_j, b_j, c_j, d_j\}_{j=1:M}] = \prod_{j=1}^M \mathbb{P}[{a_j, b_j, c_j, d_j}]$ where $\mathbb{P}[{a_j, b_j, c_j, d_j}] \sim \mathcal{N}(0,1)$
$q_\phi(\{a_j,b_j,c_j,d_j\}_{j=1:M} | \mathbf{r}_{i,1:M}) = q_\phi(\{a_j,b_j,c_j,d_j \}_{j=1:M}) = \prod_{j=1}^M q_\phi(\{a_j,b_j,c_j,d_j \}_{j=1:M})$
The goal of VIBO is to pick a family of distribution that “best approximates the true posterior by minimizing an estimate of mismatch between true and approximate distributions.” Observed variable $x \in \mathcal{X}$ represents responses from a single student $\mathbf{r}_i$ and latent variables $z \in \mathbf{\mathcal{Z}}$ represents ability and item characteristics $\theta_i, \{a_j,b_j,c_j,d_j\}_{j=1:M}$.
\[q_{\psi^*(x)}(z)=\arg\min_{q_{\psi(x)}} D_{\text{KL}}\big(q_{\psi(x)}(z),|,p(z|x)\big)=\arg\max_{\psi(x)}\mathbb{E}{q_{\psi(x)}(z)}\left[\log\frac{p(x,z)}{q_{\psi(x)}(z)}\right]\]by definition: minimixing the KL-divergence to the log posterior is eqivalent to maximizing the ELBO
Let $\mathbb{P}_D(x)$ be an empirical distribution over the observed variables. The average quality of the variational approximations is
\[\mathbb{E}_{\mathbb{P}_D(x)}\left[\max_{\psi(x)} \mathbb{E}_{q_{\psi(x)}(z)}\left[\frac{\mathbb{P}[x,z]}{q_{\psi(x)}(z)}\right]\right]\]Learning an approximate posterior for each $x \in D$ can grow to be unweildy in a large dataset. Wu et al. (2020) proposed an amortized function $f_\phi$ (parameterimized by $\phi$) that maps a person’s responses directly to parameters of their approximate posterior distribution. The number of parameters in amortization is vastly smaller than learning a per-observation posterior.
\[\max_\phi \mathbb{E}_{\mathbb{P}_D(x)}\left[\mathbb{E}_{q_\phi(z|x)}\left[\log \frac{\mathbb{P}[x,z]}{q_\phi(z|x)}\right]\right]\]Wu et al. (2020) proved that VIBO is a lower bound on the log marginal probability of person $i$’s response.
\[\begin{flalign} \log \mathbb{P}_\theta[\mathbf{r}_{i,1:M}] \ge \text{VIBO} &\triangleq \mathcal{L}_{\text{recon}} + \mathbb{E}_{q_\phi(\{a_j,b_j,c_j,d_j\}_{j=1:M}|\mathbf{r}_{i,1:M})}[D_{\text{ability}}] + D_{\text{item}} \\ \mathcal{L}_{\text{recon}} &= \mathbb{E}_{q_\phi(\theta_i, \{a_j, b_j, c_j, d_j\}_{j=1:M}|\mathbf{r}_{i,1:M})}\left[\log \mathbb{P}_\theta[\mathbf{r}_{i,1:M}|\theta_i, \{a_j, b_j, c_j, d_j\}_{j=1:M}]\right] \\ D_{\text{ability}} &= D_{\text{KL}}\left(q_\phi(\theta_i|\{a_j,b_j,c_j,d_j\}_{j=1:M}, \mathbf{r}_{i,1:M})\|\mathbb{P}[\theta_i]\right)\\ D_{\text{item}} &= D_{\text{KL}}(q_\phi(\{a_j,b_j,c_j,d_j\}_{j=1:M}|\mathbf{r}_{i,1:M})\|\mathbb{P}[\{a_j,b_j,c_j,d_j\}_{j=1:M}])\\ \end{flalign}\]and estimated the gradients with respect to $\theta$ and $\phi$.
\[\begin{flalign} \nabla_\theta\text{VIBO} &= \nabla_\theta\mathcal{L}_{\text{recon}} \\ &= \mathbb{E}_{q_\phi(\theta_i, \{a_j,b_j,c_j,d_j\}_{j=1:M})}[\nabla_\theta\log\mathbb{P}_\theta[\mathbf{r}_{i,1:M}|\theta_i, \{a_j,b_j,c_j,d_j\}_{j=1:M}]] \\ \nabla_\phi\text{VIBO} &= \nabla_\phi \mathbb{E}_{q_\phi(\{a_j,b_j,c_j,d_j\}_{j=1:M}|\mathbf{r}_{i,1:M})}[D_\text{ability}] + \nabla_\phi D_{\text{item}} \\ &= \nabla_\phi\mathbb{E}_{q_\phi(\theta_i, \{a_j,b_j,c_j,d_j\}_{j=1:M})} \left[\frac{\mathbb{P}[\theta_i]\mathbb{P}[\{a_j,b_j,c_j,d_j\}_{j=1:M}]}{q_\theta(\theta_i, \{a_j,b_j,c_j,d_j\}_{j=1:M}|\mathbf{r}_{i,1:M})}\right] \\ \end{flalign}\]Scale Linking
The summary below is pulled directly from Kim and Lee (2004) unless otherwise stated. The paper extends linkage methods to mixed-format tests (e.g. multiple choice or short essay questions). The documentation below has been modified to account for only dichotomous tests. Kim and Lee (2004) found that characteristic curve method yield more linking results than moment methods.
| Term | Definition |
|---|---|
| $A, B$ | (scale) linking coefficients slope and intercept, respectively |
| $\Psi(\theta_{\texttt{old}}), \Psi(\theta_{\texttt{new}})$ | continuous distributions of $\theta_{\texttt{old}}$ and $\theta_N$ respectively |
| $\hat T(\cdot)$ | test characteristic function (an aggregation of item characteristic curves) |
Motivation
Aside: Because the latent (ability) scale in IRT model is arbitrary, (unlinked) item and examinee parameters are identical only up to a set of linear transformations listed below (Lord, 1980; Rupp and Zumbo, 2006). Let $*$ item and examinee parameters represent “new” parameters Below is an informal proof of equivalence.
- $\theta^* = A\theta + B$
- $a^* = a/A$
- $b^* = Ab+B$
- $c^* = c$
Consider a situation in which two populations of examinees: $\texttt{old}$ and $\texttt{new}$, take a “test,” and their response data is later trained to train two separate IRT models. When item parameters are estimated separately for two groups, item parameters are expressed on different measure scales (Battauz, 2017). Before comparing item parameter estimates derived from different groups, it is necessary to “transform” them to obtain values expressed on the same “metric” (Battauz, 2017).
\[\begin{flalign} \theta_{\texttt{old}} &= A\theta_{\texttt{new}} + B && \text{$\theta_{\texttt{old}}$ and $\theta_{\texttt{new}}$ are group dependent but should be linearly related because of invariance assumption (Lord, 1980)}\\ a_{j, \texttt{old}} &= a_{j, \texttt{new}} / A && \text{$j$th item discrimination parameter on old $\theta_{\texttt{old}}$ and new scale $\theta_{\texttt{new}}$} \\ b_{j, \texttt{old}} &= Ab_{j, \texttt{new}} + B \\ c_{j, \texttt{old}} &= c_{j, \texttt{new}} && \text{$c_j, d_j$ are independent of ability $\theta$ and are not affected by the linear transformation}\\ d_{j, \texttt{old}} &= d_{j, \texttt{new}} \\ \end{flalign}\]Moment Methods
Mean/mean method \(\begin{flalign} A_{\texttt{mean/mean}} &= \frac{\texttt{mean}(\hat a_{\texttt{new}})}{\texttt{mean}(\hat a_\texttt{old})} \\ B_{\texttt{mean/mean}} &= \texttt{mean}(b_\texttt{old}) - A_{\texttt{mean/mean}}\texttt{mean}(\hat b_{\texttt{new}}) \end{flalign}\)
Mean/sigma method \(\begin{flalign} A_{\texttt{mean/sigma}} &= \frac{\texttt{standard_deviation}(\hat b_{\texttt{old}})}{\texttt{standard_deviation}(\hat b_{\texttt{new}})} \\ B_{\texttt{mean/sigma}} &= \texttt{mean}(\hat b_{\texttt{old}}) - A_{\texttt{mean/sigma}} \texttt{mean}(\hat b_{\texttt{new}}) \end{flalign}\)
Characteristic Curve Methods
To develop characteristic curve methods, the item characteristic curve or a test characteristic curve on the new (ability) scale is transformed with its “counterpart” on the old scale, and vice versa. This transformation requires item parameter estimates on one scale to be expressed as those transformed to the other; two ways of transformation —new-to-old and old-to-new—are conducted.
Such transformations vary by IRT model; item parameter estimate transformations for 3PL models are as follows
\[\begin{flalign} \texttt{new}\rightarrow\texttt{old} &: \hat a^*_{j,\texttt{new}} = \frac{\hat a_{j, \texttt{new}}}{A} \hspace{5em} \hat b^*_{j, \texttt{new}} = A\hat b^*_{j, \texttt{new}} + B \\ \texttt{old}\rightarrow\texttt{new} &: \hat a^{\#}_{j, \texttt{old}} = A\hat a_{j, \texttt{old}} \hspace{5.5em} \hat b^{\#}_{j, \texttt{old}} = \frac{\hat b_{j, \texttt{old}} - B}{A} \end{flalign}\]Robitzsch (2024) explored asymptotic bias1 of the Haebara and Stocking-Lord linking methods for the 2PL IRT model. The Stocking-Lord linking method had substantial advantages over Haebara linking in the presence of differential item functioning (described here).
Haebara method
Stocking-Lord method: The criterion function $F$ has been defined to be non-symmetric so that only the target scale (i.e. $\theta_{\texttt{old}}$ in the case of new-to-old transformations) is taken into account. It may be desirable that the transformed scale (i.e. $\theta_{\texttt{new}}$ in the case of new-to-old transformation) also be taken into account, making $F$ symmetric.
| Formulation | Approximation |
|---|---|
| $F = F_1 + F_2$ | $F \cong F^* = F_1^* + F_2^*$ |
| $F_1 = \int_{-\infty}^{+\infty} \left[\hat T(\theta_{\texttt{old}}) - \hat T^*(\theta_{\texttt{old}})\right]^2 \Psi_1(\theta_{\texttt{old}}) d\theta_{\texttt{old}}$ | $F_1^* = \frac{1}{L^*} \sum_{i=1}^{N_{\texttt{old}}} \left[ \hat T(\theta_{i,\texttt{old}}) - \hat T^{\ast}(\theta_{i, \texttt{old}}) \right]^2 W_1(\theta_{i, \texttt{old}})$ |
| $F_2 = \int_{-\infty}^{+\infty} \left[ \hat T(\theta_{\texttt{new}}) - \hat T^{\#}(\theta_{\texttt{new}}) \right]^2 \psi_2(\theta_{\texttt{new}})d\theta_{\texttt{new}}$ | $F_2^* = \frac{1}{L_2^*} \sum_{i=1}^{N_{\texttt{new}}}\left[\hat T(\theta_{i, \texttt{new}}) - \hat T^{\#}(\theta_{i, \texttt{new}})\right]^2 W_2(\theta_{i, \texttt{new}})$ |
Differential Item Functioning (DIF)
The success of IRT applications requires satisfactory fit between model and data (Kang and Chen, 2007). The most critical problem caused by model misfit may be that the hallmark feature of IRT, parameter invariance, no longer applies (Shepard, Camilli, and Williams, 1984; Bolt, 2002; Rupp and Zumbo, 2004).
DIF is a violation of the invariance assumption in IRT and occurs when the probability of a positive response for examinees at the same ability level varies in different groups (e.g. age, ethnicity, gender, education) (Battauz, 2017; Columbia University Mailman School of Public Health, n.d.).
DIF Types
| Benign DIF (Breslau et al, 2008) | Adverse DIF (Breslau et al, 2008) | |
|---|---|---|
| groups differ in their probabilities of endorsing at item because … | the item taps a dimension of the underlying ability between groups | artifactual elements in the measurement process (e.g. different understandings of a word/phrase) |
| reflects | real group differences in ability | biases in the measurement process |
There are no unambiguous quantitative methods to distinguish benign from adverse DIF (Columbia University Mailman School of Public Health, n.d.). Qualitative methods, such as focus groups, interviews, and careful revies of items by content experts, can minimize the degree of differential item functioning (Columbia University Mailman School of Public Health, n.d.).
Methods of Quantifying DIF
If the parameter values or item response functions are identical in both populations, then the probabilities of responding to a category are the same and therefore the item is DIF-free (Wells, 2021).
non-DIF aggregate metrics: can be used regardless of similar abilities between examinees. Selçuk and Demir, 2024 and Weber et al (2026) employed RMSE and/or MAE and had a ground truth estimate of item and (sometimes) model parameters.
\[\begin{flalign} \text{RMSE} &= \sqrt{\frac{1}{M} \sum_{j=1}^M \left(\hat b_{j, \texttt{old}} - \hat b_{j, \texttt{new}}\right)^2} \\ \text{MAE} &= \frac{1}{M} \sum_{j=1}^M \left|\hat b_{j, \texttt{old}} - \hat b_{j, \texttt{new}}\right| \\ & \text{for an arbitrary item parameter (e.g. $b_j$)} \end{flalign}\]DIF per-item metrics. Determines whether item $j$ can be marked as DIF Kim and Cohen (2009) compared Lord’s $\mathcal{X}^2$, Raju’s area measures and the likelihood ratio test on a university mathematics placement test and found that there was close agreement among the three DIF detection procedures. However, Lord’s $\mathcal{X}^2$ statistic was more effective at detecting simulated DIF than the two Z-tests of Raju’s area Kim and Cohen (2009). The Z test for the exact signed area was the least effective and was most likely to result in false negative errors Kim and Cohen (2009).
Plotting item parameter drift (Rupp and Zumbo, 2006): item parameters drift consitutes a lack of invariance at the individual item level or item set level. When lack of invariance effects are expressed with respect to response probabilities, it appears quite complex because of the curvature and asymptotic behavior of the ICCs.
\[\Delta_{ij} = \mathbb{P}_j[\mathbf{r}_{ij}=1|\theta_i, a_j, b_j, c_j, d_j] - \mathbb{P}_j[\mathbf{r}_{ij}=1|\theta_i', a_j', b_j', c_j', d_j']\]It is possible to plot the $\Delta_{ij}$ values to illustrate this complex behavior more closely.
where $\texttt{Alpha}$ refers to $a_j$, $\texttt{Delta}$ refers to $\Delta_{ij}$, and $\texttt{Theta - Beta}$ refers to $\theta_i - b_j$. This allow researchers to quickly gauge the severity of the lack of invariance effects on the logit and probability scales and from a practical viewpoints, allows one to compute and visualize different lack of invariance effects on response probabilities and examinee true scores.
Lord’s $\mathcal{X}^2$ test (Lord, 1980): Note that $^*$ refers to the scale linked version. Formulation follows Battauz (2017)
\[H_0: v_{j, \texttt{old}} = v^*_{j, \texttt{new}} \equiv \begin{pmatrix} a_{j, \texttt{old}} \\ b_{j, \texttt{old}} \\ c_{j, \texttt{old}} \end{pmatrix} = \begin{pmatrix}a^*_{j, \texttt{new}} \\ b^*_{j, \texttt{new}} \\ c^*_{j, \texttt{new}}\end{pmatrix}\]\(\mathcal{X}_j^2 = \left(v_{k,\texttt{old}} - v^*_{j,\texttt{new}}\right)^T\left(\Sigma_{j, \texttt{old}}+\Sigma_{j, \texttt{new}}\right)^{-1}\left(v_{j,\texttt{old}}-v_{j, \texttt{new}}\right)\) where $\Sigma_{j, \texttt{old}}$ and $\Sigma_{j, \texttt{new}}$ represent the estimate covariance matrix for $v_{j, \texttt{old}}$ and $v_{j, \texttt{new}}$ respectively.
NLP Applications
Lalor et al (2020) experimented on Natural Language Inference and Sentiment Analysis to show that the likelihood of answering a question correctly is impacted by the question difficulty. Logistic regression models used training set size and item difficulty as parameters to predict whether a deep neural network would label an item correctly. Performance starts to look more human; as DNNs are trained with more data, “easy” examples are learned more quickly than hard examples.
Spurious Correlations
Models can achieve strong results by simply using the hypothesis of a premise-hypothesis pair and ignoring the premise entirely (Gururangan et al, 2018, Tsuchiya (2018), Poliak et al (2018))
Citations
Battauz (2017). On Wald Tests for Differential item Functioning Detection ↩
Bolt. (2002) A Monte Carlo Comparison of Parametric and Nonparametric Polytomous DIF Detection Methods
Breslau et al. (2008) Differential item functioning between ethnic groups in the epidemiological assessment of depression
Columbia University Mailman School of Public Health. (n.d.) Differential Item Functioning ↩
Columbia University Mailman School of Public Health. (n.d.). Item Response Theory.
Gururangan et al (2018). Annotation Artifacts in Natural Language Inference Data
Kang and Chen (2007). An Investigation of the Performance of the Generalized S-$X^2$ Item-Fit Index for Polytomous IRT Models
Kim and Cohen (2009). A Comparison of Lord’s Chi-Square, Raju’s Area Measures, and the Likelihood Ratio Test on Detection of Differential Item Functioning
Kim and Lee (2004). IRT Scale Linking Methods for Mixed-Format Tests
Lalor et al (2016). Building an Evaluation Scale using Item Response Theory
Lalor et al (2020). Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study
Lord (1980) Applications of item response theory to practical testing problems.
Poliak et al (2018). Hypothesis Only Baselines in Natural Language Inference
Robitzsch (2024). Bias-reduced Haebara and Stocking-Lord Linking in the Presence of Differential Item Functioning
Raju. (1988) The area between two item characteristic curves
Rupp and Zumbo. (2004) A note on how to quantify and report whether IRT parameter invariance holds: When Pearson correlations are not enough
Rupp and Zumbo. (2006) Understanding Parameter Invariance in Unidimensional IRT Models
Selçuk and Demir (2024) Comparison of item response theory ability and item parameters to classical and Bayesian estimation methods
Shepard, Camilli, and Williams, (1984). Accounting for Statistical Artifacts in Item Bias Research
Tsuchiya (2018). Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment
Weber et al. (2026) The stability of IRT parameters under several test equating conditions ↩
Wells (2021). Methods Based on Item Response Theory
Wu et al (2020). Variational Item Response Theory: Fast, Accurate, and Expressive
-
Asymptotic bias refers to the error between an expected statistical estimate and the true population value when the sample size grows infinitely large. ↩