Improve your assessments by making them adaptive!
Computerized adaptive testing (CAT) is a sophisticated approach for delivering tests, surveys, and other measurements. It is based on complex computer algorithms that adapt the test to each examinee while also controlling for practical issues such as content distribution, item exposure, and test length. CATs have been shown to reduce test length by up to 90% without a loss of precision. However, to achieve such a reduction and make use of the advantages of CAT, it is absolutely essential that the CAT developer perform research studies to simulate the performance of CATs (see Thompson & Weiss, 2011). CATSim is the only software program that completely fills this role.
Establish validity with simulation studies
CATSim implements three types of simulations for computerized adaptive testing (CAT): post-hoc (real-data) simulations, hybrid simulations, and monte-carlo simulations. In implementing a CAT program, all three types of simulation can be used at various stages of the CAT development process. CATSim options allow you to implement all three types of simulations, varying CAT starting thetas, theta estimation methods, item selection methods, item exposure controls, and termination criteria. CATSim will implement simulations for item banks of up to 999 items, with no limit on the number of examinees for both post-hoc and hybrid simulations, and a limit of 10,000 examinees for monte-carlo simulations. However, CAT simulations can be done with as few as 200 examinees or fewer if they adequately represent the population to which the CAT will be applied. CATSim implements simulations for all three dichotomous item response theory (IRT) models and five polytomous IRT models. CATSim includes all of the CAT options in FastTest, our secure platform for online testing, so that the results of using CATSim can easily be implemented in your testing program.
CATSim was previously known as POSTSIM 2.0. Enhancements in the current version include:
- Addition of monte-carlo and hybrid simulations
- Five polytomous models: graded, generalized rating scale, Rasch rating scale, Rasch partial credit, generalized partial credit
- Constrained CAT options: content balancing, exposure control, enemy items
- Beta family of distributions for monte-carlo simulations
- Expanded termination options
- Multiple termination criteria
- Weighted maximum likelihood theta estimation
- Expanded output options
CATSim is designed to present options that directly represent specifications that must be made for CAT delivery. CAT requires the specification of 5 components.
Use parameters from your own bank, or in monte-carlo simulation generate a new bank: IRT parameters can be fixed to desired values, read from a file, or randomly generated from a wide variety of distributions.
You can then simulate CATs under a range of options:
- Initial theta: fixed, randomized, or read from existing data.
- Three primary item selection algorithms, with subalgorithms representing practical concerns such as content distribution, item exposure, and enemy items.
- Scoring: maximum likelihood, weighted maximum likelihood, or Bayesian (EAP or MAP) estimation.
- Termination criteria, such as target SEM or information.
CATSim records the simulations with item-by-item detail, but also provides a summary report describing the dependent variables necessary to adequately evaluate a CAT, such as average test length, accuracy, and item exposure.