Entries by Nathan Thompson, PhD

Bellezza & Bellezza (1989): Error Similarity Analysis

This index evaluates error similarity analysis (ESA), namely estimating the probability that a given pair of examinees would have the same exact errors in common (EEIC), given the total number of errors they have in common (EIC) and the aggregated probability P of selecting the same distractor.  Bellezza and Bellezza utilize the notation of k=EEIC […]

Frary, Tideman, Watts (1977): g2 collusion index

The Frary, Tideman, and Watts (1977) g2 index is a collusion (cheating) detection index, which is a standardization that evaluates a number of common responses between two examinees in the typical standardized format: observed common responses minus the expectation of common responses, divided by the expected standard deviation of common responses.  It compares all pairs […]

Wollack 1997 Omega Collusion Index

Wollack (1997) adapted the standardized collusion index of Frary, Tidemann, and Watts (1977) g2 to item response theory (IRT) and produced the Wollack Omega (ω) index.  It is clear that the graphics in the original article by Frary et al. (1977) were crude classical approximations of an item response function, so Wollack replaced the probability […]

Wesolowsky (2000) Zjk collusion detection index

Wesolowsky’s (2000) index is a collusion detection index, designed to look for exam cheating by finding similar response vectors amongst examinees. It is in the same family as g2 and Wollack’s ω.  Like those, it creates a standardized statistic by evaluating the difference between observed and expected common responses and dividing by a standard error.  […]

Response Time Effort

Wise and Kong (2005) defined an index to flag examinees not putting forth minimal effort, based on their response time.  It is called the response time effort (RTE) index. Let K be the number of items in the test. The RTE for each examinee j is where TCji is 1 if the response time on […]

Holland K Index and K Variants for Forensics

The Holland K index and variants are probability-based indices for psychometric forensics, like the Bellezza & Bellezza indices, but make use of conditional information in their calculations. All three estimate the probability of observing  wij  or more identical incorrect responses (that is, EEIC, exact errors in common) between a pair of examinees in a directional […]

What is a Psychometrician?

A psychometrician is someone who studies the process of assessment, namely how to develop and validate exams, regardless of the type of assessment (certification, employment, university admissions, K-12, etc.).  They are familiar with the scientific literature devoted to the development of fair, high-quality assessments, and they use this knowledge to improve assessments.  They implement aspects […]

,

What is an item distractor?

An item distractor, also known as a foil or a trap, is an incorrect option for a selected-response item on an assessment. What makes a good item distractor? One word: plausibility.  We need the item distractor to attract examinees.  If it is so irrelevant that no one considers it, then it does not do any […]

, ,

What are technology enhanced items?

Technology-enhanced items are assessment items (questions) that utilize technology to improve the interaction of the item, over and above what is possible with paper.  Tech-enhanced items can improve examinee engagement (important with K12 assessment), assess complex concepts with higher fidelity, improve precision/reliability, and enhance face validity/sellability.  To some extent, the last word is the key […]

, ,

What is automated item generation?

Automated item generation (AIG) is a paradigm for developing assessment items (test questions), utilizing principles of artificial intelligence and automation. As the name suggests, it tries to automate some or all of the effort involved with item authoring, as that is one of the most time-intensive aspects of assessment development – which is no news […]