Linear on the fly testing (LOFT) is an approach to delivering assessments to examinees. In general, there are two families of test delivery. Static approaches deliver the same test form or forms to everyone; this is the ubiquitous and traditional “linear” method of testing. Algorithmic approaches deliver the test to each examinee based on a computer algorithm; this includes LOFT, computerized adaptive testing (CAT), and multistage testing (MST).
What is linear on the fly testing?
The purpose of linear on the fly testing is to give every an examinee a linear form that is uniquely created for them – but each one is create to be psychometrically equivalent to all others to ensure fairness. For example, we might have a pool of 200 items, and every person only gets 100, but that 100 is balanced for each person. This can be done be ensuring content and/or statistical equivalency, as well ancillary metadata such as item types or cognitive level.
This portion is relatively straightforward. If your test blueprint calls for 20 items in each of 5 domains, for a total of 100 items, then each form administered to examinees should follow this blueprint. Sometimes the content blueprint might go 2 or even 3 levels deep.
There are, of course, two predominant psychometric paradigms: classical test theory and item response theory. With CTT, forms can easily be built to have an equivalent P value, and therefore expected mean score. If point-biserial statistics are available for each item, you can also design the algorithm to design forms that have the same standard deviation and reliability. With item response theory, the typical approach is to design forms to have the same test information function, or inversely, conditional standard error of measurement function. To learn more about how these are implemented, download our IRT Scoring Spreadsheet or Classical Form Assembly Tool.
LOFT is typically implemented by publishing a pool of items with an algorithm to select subsets that meet the requirements. Therefore, you need a psychometrically sophisticated testing engine that stores the necessary statistics and item metadata, lets you define a pool of items, specify the relevant options such as target statistics and blueprints, and deliver the test in a secure manner. Very few testing platforms can implement a quality LOFT assessment.
Why all this?
It certainly is not easy to build a strong item bank, design LOFT pools, and develop a complex algorithm that meets content and statistical balancing needs. So why would an organization use linear on the fly testing? Well, it is much more secure than having a few linear forms. Since everyone receives a unique form, it is impossible for word to get out about what the first questions on the test are. And of course, we could simply perform a random selection of 100 items from a pool of 200, but that would be potentially unfair. Using LOFT will ensure the test remains fair and defensible.
Latest posts by Nathan Thompson, PhD (see all)
- Is R for psychometrics finally becoming mainstream? - April 22, 2019
- Flagging invalid scores with Time-Score analysis - April 22, 2019
- Is teaching to the test a bad thing? - March 7, 2019