What is automated test assembly?
Just what it sounds like – provide the software a list of items in your pool and relevant metadata, then tell it to assemble 4 forms of 100 items with content coverage that meets your blueprints, 20 items overlap, and an average P value of 0.72… DONE. It only takes seconds.
TestAssembler – Automated Test Assembly with Anchor Blocks
TestAssembler is a simple, effective tool for automated test assembly (form building or construction) using either classical test theory (CTT) or item response theory (IRT). TestAssembler was designed with one purpose in mind: to save you time and money.
Consider the time that it would take to build 8 test forms from a bank of item statistics where each form
- is equivalent in terms of content distribution and psychometric statistics;
- has internal anchor blocks that are also equivalent in terms of content distribution and psychometric statistics;
- is constrained with additional bounds on the statistics of individual items.
To do this manually would take you hours or even days. TestAssembler can complete this process in seconds and do it more accurately – meaning that TestAssembler can literally pay for itself with a single use!
Automated test assembly (ATA) is widely regarded as a valid and efficient method to the practical problem of creating test forms that are parallel and/or linked via anchor items (Davey and Hendrickson, 2010). TestAssembler is the first publicly available program designed specifically to implement ATA, bringing the power of ATA to every testing organization. It can apply both classical test theory (CTT) and item response theory (IRT).
What does TestAssembler do?
- Produces a high-quality Rich Text File (.rtf) report complete with title page, colored headings, tables, narrative, and explanations of the results
- For CTT assembly: Estimates the mean and standard deviation of raw (number-correct) scores on each form
- For IRT assembly: Provides plots of the test response function(s) and item information function(s) separately for each form
- Allows you to specify a higher minimum discrimination statistic for anchor items, so that better items are selected as anchors
- Simultaneously build up to 9 forms
- For CTT tests there are two options for the psychometric targets of both the internal anchor block and the total form: mean difficulty (P) and a grouped frequency distribution of P
- For IRT tests you can specify a psychometric target based on the mean difficulty (b) parameter
- Allows for item banks up to 1,000 items
- Input of item bank statistics is by a simple text file with item statistics and content codes
- Includes a detailed manual that explains the input, output, and operation of the program
Automate the drudgery of form assembly
TestAssembler’s interface (see screenshot below) is straightforward and user-friendly. It is divided into three tabs:
- Input Options – Specify the input file, the output file, and the run title to appear in the output. In addition, you must specify the measurement model (either CTT or IRT) used for test assembly. You can also control the range of acceptable item statistics, such as classical difficulty values between 0.50 and 0.90; only items that meet your specifications will be available for selection.
- New Form Specifications – Specify the number of forms, number of items on each form, then number of content areas, the number of items in each content area, and the psychometric target for the final forms
- Anchor Items – Specify the percentage of the total form to be an internal anchor block, as well as its statistical characteristics
Then click “Run” and your new forms are created in just a few seconds!
Use the output to build defensible test forms
The report produced by TestAssembler provides:
-A description of the program and the purpose of the report
-A record of the user specifications so that the analysis can be repeated in the future
-A description of the item bank status with respect to the build
-A list of the anchor items selected as well as a description of the rationale for anchor items
-A list of all items on the test, including their domain and item statistics, allowing you to examine the results of the build immediately
-For CTT tests: Summary statistics of the build, including the average difficulty, average discrimination, and projected raw score average and standard deviation.
-For IRT tests: Summary statistics for the item parameters, including the average item difficulty (b), average item discrimination (a) for the 2- and 3-parameter models, average pseudo-guessing parameter (c) for the 3-parameter model.