| POSTSIM - Post-Hoc (Real-Data) Simulation of Computerized Adaptive Testing |
POSTSIM 2.0 implements post-hoc or "real-data" simulations of computerized
adaptive testing. In implementing a computerized adaptive test (CAT) testing
program, post-hoc simulation is an important step prior to live implementation
of a CAT. Post-hoc simulation allows you to evaluate various CAT testing parameters
prior to live testing, including entry points into the CAT, item selection
options, scoring methods, and termination criteria, so that your live CAT will
function optimally with the item bank that you have calibrated using dichotomous
item response theory (IRT) models. POSTSIM includes all of the CAT options
in the FastTEST Professional
Testing System, as well as some other CAT options not yet available in FastTEST Pro.
Whereas monte carlo simulations are typically useful in the early stages of investigating the performance characteristics of CAT procedures,
post-hoc simulation is particularly useful when a calibrated item bank is available, prior to the implementation of a live CAT. In contrast to monte carlo simulations, post-hoc simulations use the responses of real examinees to real items.
To implement a post-hoc simulation, you will need item response data from a group of examinees who have answered a set of multiple-choice items administered as a conventional test. In the case of CAT, this test would typically consist of the IRT-calibrated items that will comprise the CAT item bank. The post-hoc simulation then "re-administers" those items to the examinees (using the responses they have already provided) "as if" the item bank had been administered using various CAT procedures.
POSTSIM will implement simulations for item banks of up to 999 items, with no limit on the number of examinees.
Input requirements include a data file in the same format used in our Item and Test Analysis Package with ITEMAN 3.6, RASCAL, and/or XCALIBRE, and a file of IRT item parameter estimates from a 1-, 2-, or 3-parameter logistic model (.g., an XCALIBRE output .PAR file).
CAT Options in POSTSIM 2
Initial q
POSTSIM provides three options for beginning your CAT:
The first option allows you to begin a simulated CAT with a constant prior q
value of your choosing for all examinees. The second option allows your prior q
to randomly vary within an interval you specify. The third option allows you to specify a different prior theta (and, optionally its standard error) for each examinee.
q
Estimation
POSTSIM provides two ways to estimate q
: Maximum likelihood and Bayesian (EAP or MAP)
For maximum likelihood estimation, when a q
estimate cannot be calculated (typically at the beginning of a CAT), you can use either a fixed step size of your choosing, or temporarily estimate q using Bayesian estimation until a mixed response pattern has been obtained.
Item Selection Options
POSTSIM provides three item selection options:
The first is maximum Fisher information item selection. The second item selection option provides a means of controlling the exposure of early items in a CAT. The third option provides the capability of implementing sequential (rather than adaptive) testing. In sequential testing, items are ordered by maximum information at the specified q value and are administered in that order.
Termination Options
POSTSIM provides a number of different options for terminating a CAT:
The first option allows you to administer a fixed-length CAT. When a fixed-length CAT is used, standard errors will likely vary across examinees.
The second and third options allow you to control the standard error of the q
estimate (observed SEM), resulting in CATs that measure each examinee to a prespecified SEM, or "equiprecise" measurements.
The fourth and fifth options allow you to terminate the CATs when the information in the next item to be administered falls below a value you specify, with or without a minimum number of items. It is particularly useful for tests that have information functions that are not approximately horizontal, which is the ideal for many applications of CAT.
The sixth and seventh termination options are used with adaptive mastery/classification testing where a cutoff value on q
is specified and the CAT is designed to classify individuals as above or below the cutoff value. The options allow you vary the width of the confidence interval (in SEMs) around the estimated q
for each examinee, and to require a minimum number of items.
The last termination criterion will administer all the items in the test as a CAT.
Using the Minimum Number of Items check box in conjunction with any of the variable termination options (all except the first and last options), you can require a minimum number of items to be administered before the termination criterion is applied.
Output Files
POSTSIM creates three types of output files, some of which are optional: basic output files, user-named standard output files, and user-named optional output files.
Basic Output Files
Two standard output files are created for each run:
- The item-by-item information table for the items using q
values from -3.0 to +3.0 in increments of 0.05. Item information values are sorted from highest to lowest at each value of q.
- Item numbers corresponding to the sorted information values in the information table.
User-Named Output Files
The following files are produced for each run:
- A summary file that includes , plus the following additional information :
- A file with the summary data for each examinee in a form that can be imported into a spreadsheet or statistical software for further
analysis,
- A file with the final CAT q estimates and their standard errors for each examinee, one line per examinee. If you have more than one test per examinee, these q values (and, optionally, their standard errors) can then be used as starting q estimates for another test.
-
User-Named Optional Output Files
Four optional files can be created for each run. These are comma-separated-values (CSV) files with one line per examinee. They are designed to be imported into a spreadsheet or statistical software for further analysis. The entries in these files are the item-by-item values of the following variables after each item in the CAT, with one line of entries for each examinee:
- CAT q estimates.
- SEMs associated with each q estimate.
- Item responses scored as correct (1) or incorrect (0).
- The item number of the item administered.
Sample Data Files
POSTSIM 2 is supplied with a sample data file and two sample item parameter files – one with item parameters only and one XCALIBRE parameter output file. Sample output files are also provided for two types of CAT.
System Requirements
Windows 95, 98, NT, ME, 2000, XP, Vista
Free Demo/Student Version and Manual
- A free fully-functioning demo/student version is available.
- This version has all the functions of POSTSIM 2, except that it is limited
to 50 examinees and 50 items.
- This version can be converted to the commercial
version by email, fax, or phone after payment arrangements have been completed.
- The POSTSIM manual is supplied as a PDF file.
Other Software for Adaptive Testing
Books on Adaptive Testing and Computerized Testing
|
To Purchase This Product
|
 |
| |
Products
Your Account
Your cart
