Getting started with regDIF

Using LASSO regularization to evaluate measurement bias in a 2-parameter logistic IRT model.

In this example, data in ida were generated to mimic an integrative data analysis, where data are pooled across multiple studies and the measurement model is evaluated for between-study and within-study (e.g., gender, age) measurement bias. These data include 6 item responses (binary) and 3 background characteristics – namely, age (continuous, centered), gender (categorical, effect-coded), and study (categorical, effect-coded).

DIF was generated to be on items 2 (age, gender, study), 3 (age, gender, study), 4 (age), and 5 (gender, study), for both intercepts and slopes.

library(regDIF)
head(ida)

##   item1 item2 item3 item4 item5 item6 age gender study
## 1     0     0     0     0     0     0  -2     -1    -1
## 2     0     0     0     0     0     0   0     -1    -1
## 3     0     0     0     0     0     0   3     -1    -1
## 4     0     1     1     1     1     1   1     -1    -1
## 5     0     0     0     0     0     0  -2     -1    -1
## 6     1     0     0     0     0     0   1     -1    -1

The item response data must first be separated from the predictor data (background variables) before running regDIF(). A single value of the tuning parameter, tau = 2, is then fit to the data.

item.data <- regDIF::ida[,1:6]
pred.data <- regDIF::ida[,7:9]
fit <- regDIF(item.data, pred.data, tau = 2)

summary(fit)

## Call:
## regDIF(item.data = item.data, pred.data = pred.data, tau = 2)
## 
## Optimal model (out of 1):
##      tau      bic 
##    2.000 4133.959 
## 
## Non-zero DIF effects:

The summary() function shows that no DIF effects remain in the model. Only the latent variable parameters and base item parameters, which were not penalized at all, are estimated to be non-zero. This is shown by using the coef method.

coef(fit)

## $tau
## [1] 2
## 
## $impact
##                [,1]
## mean.age     0.7059
## mean.gender -0.1744
## mean.study   0.9611
## var.age      0.4361
## var.gender  -0.1017
## var.study   -0.1221
## 
## $base
##               [,1]
## item1.int. -0.7385
## item2.int. -1.0813
## item3.int. -1.3993
## item4.int. -2.1289
## item5.int. -1.5276
## item6.int. -2.3278
## item1.slp.  0.9842
## item2.slp.  0.9637
## item3.slp.  0.9215
## item4.slp.  1.3943
## item5.slp.  1.1719
## item6.slp.  1.3460
## 
## $dif
##                  [,1]
## item1.int.age       0
## item1.int.gender    0
## item1.int.study     0
## item2.int.age       0
## item2.int.gender    0
## item2.int.study     0
## item3.int.age       0
## item3.int.gender    0
## item3.int.study     0
## item4.int.age       0
## item4.int.gender    0
## item4.int.study     0
## item5.int.age       0
## item5.int.gender    0
## item5.int.study     0
## item6.int.age       0
## item6.int.gender    0
## item6.int.study     0
## item1.slp.age       0
## item1.slp.gender    0
## item1.slp.study     0
## item2.slp.age       0
## item2.slp.gender    0
## item2.slp.study     0
## item3.slp.age       0
## item3.slp.gender    0
## item3.slp.study     0
## item4.slp.age       0
## item4.slp.gender    0
## item4.slp.study     0
## item5.slp.age       0
## item5.slp.gender    0
## item5.slp.study     0
## item6.slp.age       0
## item6.slp.gender    0
## item6.slp.study     0

Now that the data have been properly specified in regDIF, a more thorough investigation of DIF is warranted. The regDIF() function defaults to estimating 100 values of the tuning parameter, starting with a value large enough to penalize all DIF effects to zero. However, for brevity, only 10 values of tau are specified with the num.tau argument and we reduce the tolerance for convergence using the control argument.

fit2 <- regDIF(item.data, pred.data, num.tau = 10, control = list(tol = 1e-3))

fit2

## Call:
## regDIF(item.data = item.data, pred.data = pred.data, num.tau = 10, 
##     control = list(tol = 0.001))
## 
## regDIF results:
##           tau      bic
## 1  0.59206560 4133.065
## 2  0.41582659 4104.552
## 3  0.27857133 4087.552
## 4  0.17542684 4081.078
## 5  0.10152016 4093.533
## 6  0.05197832 4120.268
## 7          NA       NA
## 8          NA       NA
## 9          NA       NA
## 10         NA       NA

By printing the model object, 10 rows of results appear, one for each value of the tuning parameter. The first thing to notice is that 4 rows are missing. This occurs because regDIF() automatically stops model-fitting when a small value of tau would produce a non-identified model. In focusing attention to the BIC column, it is evident that the smallest value occurs well before the model would be non-identified. This is an encouraging result. The summary() function may be used again, which, with multiple values of the tuning parameter fit to the data, produces non-zero DIF effects corresponding to the model with the minimum value of BIC.

summary(fit2)

## Call:
## regDIF(item.data = item.data, pred.data = pred.data, num.tau = 10, 
##     control = list(tol = 0.001))
## 
## Optimal model (out of 10):
##          tau          bic 
##    0.1754268 4081.0782000 
## 
## Non-zero DIF effects:
##    item4.int.age    item5.int.age item5.int.gender  item5.int.study 
##           0.2163          -0.0885          -0.5713           0.6024 
##  item4.slp.study item5.slp.gender 
##          -0.0928          -0.1771

A plot of the regularization path also shows the remaining DIF effects.

plot(fit2)

To produce other model results, the fit2 object contains lists of the impact (latent variable) parameters, base (intercept and slope) item parameters, and DIF parameters for all values of the tuning parameter. For instance, the impact parameters are printed below.

fit2$impact

##                [,1]    [,2]    [,3]    [,4]    [,5]    [,6] [,7] [,8] [,9]
## mean.age     0.7042  0.7085  0.6934  0.7107  0.7693  0.9216   NA   NA   NA
## mean.gender -0.1739 -0.1188 -0.0875 -0.0776 -0.0860 -0.1113   NA   NA   NA
## mean.study   0.9585  0.9080  0.8750  0.8802  0.9030  0.9571   NA   NA   NA
## var.age      0.4317  0.4433  0.4299  0.4359  0.4218  0.3716   NA   NA   NA
## var.gender  -0.1006 -0.0902 -0.0918 -0.0655 -0.0586 -0.0219   NA   NA   NA
## var.study   -0.1243 -0.1312 -0.1115 -0.0652  0.0364  0.2551   NA   NA   NA
##             [,10]
## mean.age       NA
## mean.gender    NA
## mean.study     NA
## var.age        NA
## var.gender     NA
## var.study      NA

EAP scores and standard deviations may also be produced.

lapply(fit2$eap, head)

## $scores
##            [,1]       [,2]       [,3]       [,4]       [,5]       [,6] [,7]
## [1,] -1.8130833 -1.8125184 -1.7896639 -1.7995627 -1.8526168 -2.0039661   NA
## [2,] -1.3342935 -1.3320319 -1.3196124 -1.2915327 -1.2497203 -1.1606881   NA
## [3,] -1.2256347 -1.2404596 -1.2275429 -1.1702650 -1.0116288 -0.6035783   NA
## [4,]  2.0793250  2.0691533  2.0263565  1.9878598  1.9093402  1.6925890   NA
## [5,] -1.8130833 -1.8125184 -1.7896639 -1.7995627 -1.8526168 -2.0039661   NA
## [6,] -0.3805602 -0.3535745 -0.3491745 -0.3608825 -0.3627101 -0.2775264   NA
##      [,8] [,9] [,10]
## [1,]   NA   NA    NA
## [2,]   NA   NA    NA
## [3,]   NA   NA    NA
## [4,]   NA   NA    NA
## [5,]   NA   NA    NA
## [6,]   NA   NA    NA
## 
## $sd
##           [,1]      [,2]      [,3]      [,4]      [,5]      [,6] [,7] [,8] [,9]
## [1,] 0.7966229 0.7916416 0.7898758 0.7653328 0.7362193 0.6776154   NA   NA   NA
## [2,] 0.9317806 0.9313398 0.9215595 0.8948896 0.8499689 0.7514918   NA   NA   NA
## [3,] 1.1450472 1.1535148 1.1257071 1.0902247 1.0105346 0.8287311   NA   NA   NA
## [4,] 0.6973359 0.6949156 0.6865598 0.6655699 0.6289102 0.5495122   NA   NA   NA
## [5,] 0.7966229 0.7916416 0.7898758 0.7653328 0.7362193 0.6776154   NA   NA   NA
## [6,] 0.8553530 0.8510991 0.8372854 0.8165459 0.7742955 0.6711971   NA   NA   NA
##      [,10]
## [1,]    NA
## [2,]    NA
## [3,]    NA
## [4,]    NA
## [5,]    NA
## [6,]    NA

Finally, when data include a large number of items, observations, and predictors, regDIF() can run relatively slowly. An alternative approach, which yields much faster results, is to provide an observed proxy for the latent scores. In the case of binary data, this might be sum scores. Note that using observed proxy scores is identical to performing a multivariate regression, where the item responses are regressed on the proxy scores and background variables. (The proxy scores are simultaneously regressed on the background variables as well.)

fit3 <- regDIF(item.data, pred.data, prox.data = rowSums(item.data), num.tau = 20)

summary(fit3)

## Call:
## regDIF(item.data = item.data, pred.data = pred.data, prox.data = rowSums(item.data), 
##     num.tau = 20)
## 
## Optimal model (out of 20):
##          tau          bic 
##    0.2506739 3547.6463000 
## 
## Non-zero DIF effects:
## item3.int.gender    item4.int.age    item5.int.age item5.int.gender 
##           0.1220           0.2403          -0.0205          -0.5422 
##  item5.int.study item6.int.gender item2.slp.gender  item4.slp.study 
##           0.7356           0.0031           0.1388          -0.1627 
## item5.slp.gender 
##          -0.1602

The results show more DIF on both the intercepts (Items 3 and 6) and slopes (Items 2, 3, and 5).

More modeling possibilities with regDIF.

In addition to LASSO, other penalty functions are possible with regDIF(). For instance, the elastic net penalty combines LASSO and ridge functions, which is useful when many correlated predictors are evaluated for DIF. The elastic net is controlled by a second tuning parameter, alpha, and defaults to alpha = 1, corresponding to the LASSO penalty. In contrast, alpha = 0 corresponds to the ridge penalty. When alpha is between 0 and 1, however, the elastic net is used to perform DIF selection. For brevity, observed proxy scores are used in all model fitting below.

fit_net <- regDIF(item.data, pred.data, prox.data = rowSums(item.data), num.tau = 20, alpha = .5)

summary(fit_net)

## Call:
## regDIF(item.data = item.data, pred.data = pred.data, prox.data = rowSums(item.data), 
##     num.tau = 20, alpha = 0.5)
## 
## Optimal model (out of 20):
##          tau          bic 
##    0.5605648 3568.9282000 
## 
## Non-zero DIF effects:
## item3.int.gender    item4.int.age    item5.int.age item5.int.gender 
##           0.0715           0.1703          -0.0958          -0.3511 
##  item5.int.study item2.slp.gender  item4.slp.study    item5.slp.age 
##           0.4398           0.0814          -0.1200          -0.0006 
## item5.slp.gender 
##          -0.1406

The final elastic net results yield the same DIF effects as the LASSO results, although the amount of penalization is greater for elastic net (i.e., larger tau).

Other penalty functions include the minimax concave penalty (MCP) and the group extensions of LASSO and MCP, which penalize the intercept and slope DIF effects in tandem. The group LASSO function is shown below.

fit_grp_mcp <- regDIF(item.data, pred.data, prox.data = rowSums(item.data), num.tau = 20, pen.type = "grp.mcp")

summary(fit_grp_mcp)

## Call:
## regDIF(item.data = item.data, pred.data = pred.data, prox.data = rowSums(item.data), 
##     pen.type = "grp.mcp", num.tau = 20)
## 
## Optimal model (out of 20):
##          tau          bic 
##    0.4673465 3535.4418000 
## 
## Non-zero DIF effects:
##    item4.int.age item5.int.gender  item5.int.study    item4.slp.age 
##           0.1472          -0.5585           0.7661           0.0980 
## item5.slp.gender  item5.slp.study 
##          -0.3758           0.2438

Although the MCP results appear largely the same as the LASSO results, the group MCP function included both the intercept and slope for each background variable remaining in the final model.

In summary, the regDIF R package provides a flexible implementation of using regularization to identify DIF across multiple background characteristics.

Please reach out to [email protected] for any questions, and remember to cite regDIF in your work. Thank you kindly!

## To cite regDIF in publications use:
## 
##   Belzak, W. C. M (2021). regDIF: Regularized Differential Item
##   Functioning. version 1.0.0. Duolingo. Pittsburgh, PA.
##   https://github.com/wbelzak/regDIF/
## 
## A BibTeX entry for LaTeX users is
## 
##   @Manual{,
##     title = {regDIF: Regularized Differential Item Functioning},
##     author = {William C. M. Belzak},
##     organization = {Duolingo},
##     address = {Pittsburgh, PA},
##     note = {version 1.0.0},
##     year = {2021},
##     url = {https://github.com/wbelzak/regDIF/},
##   }